Showing posts with label big data. Show all posts
Showing posts with label big data. Show all posts

Friday, July 23, 2021

Features of NoSQL Databases and their Usage for Analytics | 5 Conceptual Questions and Answers | Concept Infographic Note


 

Topics discussed in the post the following questionnaire :

Q1) What are some predominant features of No-SQL databases ?

Q2) Why NoSQL is called "SQL with a No " with "No" as the preceding string on SQL ?

Q3) How do organisations leverage NoSQL data for their businesses ?

Q4) Where are NoSQL databases useful ?

Q5) What do you mean by the statement , "The constraints of a relational database are relaxed ? "

Difference between RDBMS and NoSQL databases | 10 Conceptual Questions and Answers | Concept Infographic Note




Questions and answers discussed in the post:
Q1) What are the different ways in which RDBMS databases differ from NOSQL databases?
Q2) What are the Applications Level feature supported by RDBMS and NoSQL Databases ?
Q3) What are the principles of RDBMS and NoSQL type databases ?
Q4) Explain about Availability aspect of RDBMS and NoSQL type databases ?
Q5) What is the data velocity supported over RDBMS and NoSQL databases ?
Q6) What do you mean by Latency . What happens if Latency is High ? And what happens if Latency is Low ?
Q7) What is the data volume supported over RDBMS and NoSQL databases ?
Q8) What are the data sources present over RDBMS and NoSQL databases ?
Q9) What are the different data types that are present over RDBMS and NoSQL ?
Q10) What are the different data accesses that are present over RDBMS and NoSQL ?
Q11) What are the different technologies that are supported over RDBMS and NoSQL ?
Q12) Write about the cost aspects of RDBMS and NoSQL





Monday, June 7, 2021

Topics for Python - Basics ( Syntax , Construct , Variables , Operators , Data Structures , Decision Making statements , Loops , Functions , Object Oriented programming )


=====================

Topics for Python - Basics

=====================

 

1)     Syntax

 *   Python is very popular because of its simplicity to use and understand

 *   Simplicity of Python programming language lies in its syntax

 *   Python does not follow any complicated formats for writing of codes

 *  Syntax of Python quite different from other languages like C, C++, Java etc and      much easier to comprehend

 

2)     Variables

 *  If one is familiar over the manner in which variables can be declared over any Python program then the topic of manner and efficient declaration of variables can also be done efficiently

 

3)     Operators

 *   Operators are the symbols that represent a particular action or process .

*  One can explore different mathematical operators and calculations that can happen using the available operators in Python

 


4)     Data Structures

 *   Data Structures are the most important concept of any programming language

 *  Data Structures are the building blocks of any programming language as all             programming languages deal with objects creation which deal with memory      allocation , memory deallocation and memory utilisation of the variables / objects using the relevant data structures for carrying out the necessary operations that are needed to be performed as per the program that one is working upon

 *  Python has especially these four data structures that are lists , sets , dictionaries and tuples .

 *   One can get to have a good understanding of the manner in which each of the above listed data structures are put to use in advanced examples which would discuss the manner in which append , deletion or structural operations take over a given data structure .


 *  The most important concepts of Python that need to be used and understood are the following :

 

1)     Decision Making Statements :

 

If , If-else , Nested If else , chained conditionals of the else if type of statements , single statement conditions all come under the category of decision-making statements . Along with earning , one may also have to practice 10+ coding problems on a daily basis which would help someone get familiar with the  manner in which one can scale up on the level of complexity of the programs and their creation

 

2)     Loops :

 

Loops are the structures that repeat a sequence of instructions . Python deals with while loops , for loops and nested loops . Along with these one may also learn loop control statements which show the manner in which loops can be initialised , implemented , interjected and put to end in the desired way of the programmer .

 

3)     Functions in Python

 Functions are the smaller pieces of code that perform some desired task . Python has four different types of functions that are User Defined functions , Built-In functions , Lambda functions and Recursion functions mainly .

 

We will just put a small recap over Functions in Python :

 

User Defined functions are those functions which a user  / programmer writes himself which takes programming constructs and statements to write a function which would achieve a desired result at the end of the function which may include the use of some built-in functions into the overall function code construct . 


Built-In functions are those functions which are created as a part of the programme package at the time of creation of the Program package and its default libraries which come together with the program at the beginning of the program package .


Lambda functions are also some form of user defined function , however the construct is limited to only one line and this usually does the work of iteration over the elements of the function , access the individual elements , or use the functions to iteratively append , retrieve or delete some elements over an existing structure or a dummy/temporary structure .

  

*  Along with the the basic concepts of the scripting part of the Python Language one also needs to have a good understanding of implementation as well as creation of features in Object Oriented Language features of Python :

 

1)     Polymorphism

2)     Inheritance

3)     Encapsulation

4)     Abstraction

 

All these aspects of Programming in Python and how the same are going to  be used in Python for Data Science and Machine Learning Modelling would be

discussed in further blogs . As I am getting disarrayed due to a some intermittent problems over my laptop I am not able to implement some of the important Algorithms and Modellling Processes . But soon , I hope to get these started on a better level .

 

Last modified: 21:32

Lists and Dictionaries in Python Dictionaries in Python

 When should one use dictionaries and when should one use lists in Python

 =====================================================

 *  Lists are the most widely used data when the data is stored in the form of integral indexes with some starting position to some given position within the Python list

 *  Lists are also mostly used when the data that needs to be maintained in a certain specific manner or  where the data needs to be sorted where finding  some data is only good enough when the user of the given set of code is knowing the exact index or position of the item element where the list item needs to be searched or retrieved

 *  Lists are also used when the ordering of items within the List item object is irrelevant as any given element within a list can be searched and retrieved just by using some conditional loops which will iterate over the list item object and then the item could be put to use directly without any form of replacement or elimination within the data structure

 *  One of the main highlighting important points about Lists as a data structure for item storage is that Lists can be used for storage of other lists and tuples as a data storage within it because of which the main list containing the sub-lists within it could act as a parent list object which can keep within it other child objects for storage and retrieval as well

 *   Now coming to dictionaries and what they are used for :-

 *  Dictionaries are perfect storage objects when the data is to be stored on the basis of positonal indexes coupled with an adjoining value associated for that index position . Here also the ordering of the data is not of great matter and does not need to sort the stored data . Finding any associated data of the dictionary object needs only a key value of that object 


*   Example where a List is used for data storage is an example of a shopping cart where any stored item object's storage does not matter .


*  Example where a Dictionary Item object is used as a storage object is that of a an examination roll / rank number sheet ( parikhya patra ) where people are arranged on the basis of their marks obtained in some examination from Rank 1 to the last rank in the exam . Usually the person who obtains the highest marks in a given exam is attributted the 1st rank and the first roll number for that examination is also alloted to the first rank holder after the results are out , the successive role numbers are also attributed according the net marks obtained  by an examiner in the considered case of the examination till the last rank/roll number of the examiner .

 

*  A dictionary is very useful data storage object where one may need to access a given data based on the item that is present over a given index position within the storage object of the data structure .

 *  As a measure of how important the dictionaries are - every module , class , function and method written in Python in the form of dictionaries where all the attribute names are in the form of Keys and all the attribute values there in can be found quickly .

 *  Therefore when one wants to decide upon which data structure to be used for a certain type of data object one might need to consider the best use cases of both these data structure objects : Lists and Dictionary Items .


 ======================================

To sum up once again for Lists and Dictionaries

 =====================================

 

*  Lists are ordered data structure objects that means that the program has control over the order in which the data is stored . This data is also sortable that means that the program can sort the data into whatever order when the data is needed to be fetched and retrieved , the list is also containing a list of iterable items i,e items that can be accessed one item at a time when the list object is iterated or executed within a loop .

 

*  Dictionaries are like the partially ordered data structures that can control the order or the manner in which the data items are inserted and stored within the object in the form of keys and attributes . Dictionaries are non-sortable and iterable in pairs of both keys and associated value objects .



 

 

 

 

 

 

 

 

 

Last modified: 18:49

Why is Artificial Intelligence a big fab hype in the recent IT and computing technology world - a collective article on current standpoint on AI

=====================================================

Why is Artificial Intelligence  a big fab hype in the IT and technology world

=====================================================

* The most comprehensible perception about artificial Intelligence in the current technological biome is a distopian future coupled with human-like looking and performing robots and realistic looking holograms that throw a completely realistic idea of a certain picture or scenario in which non-human like androids and droids would be acting and talking like real people and having human-  level or even superhuman intelligence and capablities for performing a task which would be much complex and strenuous to humans . This concept is still beyond the aegis of reality as of now and this term is called as "Artificial General Intelligence(AGI)" and this does not exist anywhere on earth till now as depicted by some high-level CGI based movies in their content .

 

* But what is actually existing as of now is that of a fractional level implementation of Deep Learning which is shortly known as DL which can do some small level specific tasks which are better than people doing the computing task .. one   needs to keep in mind over here is that it is only restricted to computing activity and factory work . Still is believed that there are multiple fundamental level limitations that does not properly get developed into AGI - Artificial General Intelligence (AGI) . So the next level of algorithmic data science and machine learning development is that of innovating the present technology to come up with better networks and better methods for shaping into an artificial Intelligence product .

 

* So inorder to get a bird's eye view of where the development stands at , one might need to look at the following persisting scenarios :

 1)     Where Deep Learning and Reinforcement Learning are standing today


 2)    What are the limitations of Deep Learning and Reinforcement Learning algorithms and  where does the current trend stand at ?


 3)    Underlying principles and applications of Human Neuroscience and human intelligence


4)     A possible architecture to achieve artificial general Intelligence

 

Last modified: 14:46

Machine Learning Algorithms used in solving some of the popular real world problems

 =====================================================================

Machine Learning Algorithms used in solving some of the popular

real world problems

=====================================================================

 *  The majority of all the real world problems in machine learning are classification and regression problems which are to be performed over structured data ( that means it doesn't take into account much of unstructured data into the account )  


*  In the real world machine learning environment , deep learning also plays a very shallow and limited role in the field of machine learning and therefore it is considered as one of the smallest factors that go into building of applications over machine learning


 *  One of the the mostly used algorithm to augment the model performance is XGBoost algorithm .This was created by developer and machine learning engineer Tianqi Chen who changed the face of applied machine learning which as already mentioned is the best case use of machine learning algorithms and the way augmentation of built model performances is done which come under the category of gradient boosting algorithms which also happens to work over and on top of structured data .

  

*  Therefore , gradient boosters like XGBoost are considered as the gold standard for modelling over structured data problems used in data competitions .Again it needs to be re-iterated and kept in memory eternally that almost all of the data problems use XGBoost and other gradient boosters for working over any data problem .

 

 *  It is also believed that the top cloud based companies like Google , Microsoft  and Amazon also use gradient boosters over their cloud paltform for analysis into structured data problems

 *  Google's Cloud based Machine Learning Platform is called as AutoML . In this platform and application , the AutoML tables use gradient boosters and deep learning models for hyperparameter tuning


Last modified: 14:20

Technical Data Analyst posted at HP @Bangalore on 07-06-21 - Self Analysis , Upscaling and Recommendation

Technical Data Analyst posted at HP @Bangalore on 07-06-21

=======================================================================

 *   This job position is posted for Bangalore


 *  The relevant years of experience needed for this job is around 0 to 2 years of experience

 

*  The secondary requirement for the fitting criteria is that the person should have a  Master's degree in computer science or Information systems

 

*   A candidate for the job should be able to leverage and scale open source tools .This aspect is currently I am also not sure and I am also not yet sure which  open source tools are being referred to in this , since collaboration tools are the most requisite open source tools that are common to all the people albeit their technological skills

 *   Proficient is SQL and architecting relational databases - NoSql is a plus{ SQL querying skills is a standard requirement across most of the job positions but the niche plays when it comes to the manner in which one is able to perform some of the SQL operations which comes with experience and exposure . Even I can say that , though I am good .. means good in SQL and would rate myself a mid-level person in expertise over SQL , I am not sure I would be able to handle more difficult or expert level queries without starting over from such minimum requirements .Also my work over NoSQL till now has been limited and has not been much diverse because of which NoSql and Big Data system level requirements is still a new thing for me }

 

*  Proficiency in Public Cloud Providers like Amazon AWS , Google GCP , Microsoft Azure etc is also a must { I have some experience over AWS cloud and am aware of the manner in which some of the applications work over AWS but I am yet to explore all the greater details and aspects of AWS and how big scale projects are deployed at AWS }

 

*     Experience over Industry - standard BI platforms { this necessitates a candidate to have some good amount of experience over working over Business Intelligence and Reporting Applications like Microsoft Power Bi and Tableau where one might be able to carry out and find out report generation and deployment activities in order to find out the necessary actionable items required for the specific work }

 

*   Knowledge of SAAS products and their multitenant cloud implementation { I this area I am just aware of the fundamental theorems that are under work and implementation is still yet to be done and scaled up , so this is one area that I shall again need to brush up and get my hands upon. I shall try to learn from the available online resources and try online cloud platforms over where such projects and modules are deployed}

 ===================================

 Description and Responsibilities on the job

 ===================================

 *  Analyse the business problems to analytics and there upon recommending the most appropriate methods to yield insights and results

 

*  Analyse Sales and Financial Data with Cloud and Operational data to present over to the clients to view

 

*  Data Warehousing and Reporting solutions to address the growing needs for reporting , analytics , and data requirements


 *  To automate cloud metrics to analyse the business requirements and business environments and on top of that gather business requirements and create data visualizations taking into consideration PaaS , Geographical data etc .

 

 

Last modified: 13:31