Showing posts with label technology. Show all posts
Showing posts with label technology. Show all posts

Monday, June 7, 2021

Why is Artificial Intelligence a big fab hype in the recent IT and computing technology world - a collective article on current standpoint on AI

=====================================================

Why is Artificial Intelligence  a big fab hype in the IT and technology world

=====================================================

* The most comprehensible perception about artificial Intelligence in the current technological biome is a distopian future coupled with human-like looking and performing robots and realistic looking holograms that throw a completely realistic idea of a certain picture or scenario in which non-human like androids and droids would be acting and talking like real people and having human-  level or even superhuman intelligence and capablities for performing a task which would be much complex and strenuous to humans . This concept is still beyond the aegis of reality as of now and this term is called as "Artificial General Intelligence(AGI)" and this does not exist anywhere on earth till now as depicted by some high-level CGI based movies in their content .

 

* But what is actually existing as of now is that of a fractional level implementation of Deep Learning which is shortly known as DL which can do some small level specific tasks which are better than people doing the computing task .. one   needs to keep in mind over here is that it is only restricted to computing activity and factory work . Still is believed that there are multiple fundamental level limitations that does not properly get developed into AGI - Artificial General Intelligence (AGI) . So the next level of algorithmic data science and machine learning development is that of innovating the present technology to come up with better networks and better methods for shaping into an artificial Intelligence product .

 

* So inorder to get a bird's eye view of where the development stands at , one might need to look at the following persisting scenarios :

 1)     Where Deep Learning and Reinforcement Learning are standing today


 2)    What are the limitations of Deep Learning and Reinforcement Learning algorithms and  where does the current trend stand at ?


 3)    Underlying principles and applications of Human Neuroscience and human intelligence


4)     A possible architecture to achieve artificial general Intelligence

 

Last modified: 14:46

Friday, April 30, 2021

Validation of Machine Learning Algorithms and Scenarios - A short article

 

                        Validation of Machine Learning Codes

 *  It is a widely accepted fact that just having some examples in the form of datasets and machine learning algorithm at hand does not  assure that solving  a machine learning problem is possible or the results would provide any desired solution

 

*  For example ... if one wants a computer to distinguish a photo of a dog from   a photo of a cat , one can do it with good examples of dogs and cats . One can then train a dog versus Cat classifier which is based on some machine learning algorithms that would output the probability that a given photo is that of a dog or a cat . All of the times for a set of photos resembling a given photo , the output would be in the form of a validation quantity which would be expressing some level of accuracy for a number which would reflect how well the classifier algorithm was able to perform those computations and with what level of  alacrity and accuracy . I am using the alacrity which should convey to the reader about the performnace and speed aspect of the identification process of the Machine Learning algorithm when computed upon a batch of photos for finding resemblance over a batch of photos of classes of photos by doing all forms of stucturisation like segmentation and clustering , KNN etc . And when it comes to the factor of accuracy one can think of the degree and magnitude in terms of percentage of resemblance of the referenced sample to the sample over which the matching is to be calculated .

 

*     Based on the probability which is exressed in percentage accuracy , one can  then decide whether the class ( that is if a dog or a cat) is based on the estimated probability as calculated by the algorithm .

 

*  Whenever the obtained probability or percentage would be higher for a dog , one can minimize the risk of making a supposed wrong assessment by choosing the higher chances which would be favouring the probability of finding a dog .

 

*  The greater the probability difference between the likelihood of a dog against that of a cat , the higher would be the confidence that one can have in their choices of finding any appropriate result

 

*  And in case , the probability difference between the likelihood of a dog against that of a cat , here it can be assumed that the picture of the subject is not clear   or probably the subjects in the picture bear much resemblance in features which would indirectly mean that some of the pictures of the cats are similar to that of the dogs and because of which a confusion may arise and lead to another supposition that whether the dogs are cattish in the concerned pictures .

 

*   On the point of training a classifier :

When you pose a problem and offer the examples , with each of the examples being carefully marked with the label or class that the algorithm should learn ; then the computer trains the algorithm for a while and then finally one would get a resulting model out of the training process of the model over the dataset .

 

*  Here , your computer trains the algorithm for a while and finally one would get a resulting model for the answer which provides one with an answer or probability .

 

*  Labellling is an another associated activity that can be carried out but in the end a probability is just an opportunity to propose a solution and get an answer

 

*  At such a point , one may have addressed all the issues and perhaps might guess that the work is finished , but still one may validate the results for ensuring that the results generated are first comprehensible to the human , make sure   that the user is able to have a clear understanding of the involved background processes and break-up analysis of the code and the result which can enable other readers to understand the code along with numbers

 

*  More over this would be elaborated in the forthcoming sessions / articles where we will look into the various modes in which the machine learning results could   be validated and made comprehensible to the users

 

Last modified: 16:39

Thursday, April 15, 2021

Fast Spreading Digital Adaptation in Emerging Nations and the associated "Theory of Everything" in the trade and commerce world - An article by Vivek Dash

 

Fast Spreading Digital Adaptation in Emerging Nations and the theory of Everything 

 

* Emerging economies have been struggling with growth through the 2010's and still the feeling of pessimism clouds over the present decade whether the steadfast growth trend could be seen to be going strong or whether the trend would be gauged down as it is being seen that People have accrued high amounts of debt during the ongoing pandemic which has become a detrimental factor to growth in not only emerging countries but also countries which are developed like USA , Russia , China , Japan , Germany , France , UK , Finland as questions over Growth tapping and keeping the same pace steady even during a pandemic period is of paramount importance to all

 * The answer to all the woes shrouding the question of development even during such cloudier and gloomy economic conditions, looming large and widespread is the harnessing of the Digital Revolution and Digital Adaptation. As most of the emerging nations have started adopting cutting edge technology at a much lower and lower cost which has allowed the countries to fuel domestic demand and overcome traditional obstacles to growth .

 * According to Reports , the number of smartphone users across the globe has skyrocketed from 150 million to 4 billion worldwide which approximates the total number of such people with a Smartphone device as nearly equals to or over half of the entire world's population . This has enabled most of the residents of the emerging world to access vital consumer and professional services over a phone which is no less than a computer in its kind and is much advanced than the personal computers which were there just a decade back .

 * According to reports from BCG - Boston Consulting Group , since 2014 more than 10,000 tech firms have been launched in emerging markets with half of them outside China

 * As per current trends and reports , India is having almost equal number of technology players as France and Germany and the number of such companies are growing much faster year on year .

* As per some key metrices and data collected by popular editorials , the onslaught and adaptation of digital revolution is much more in emerging economies than that of developed nations and has been growing at a much higher rate year on year .

 * Another report collected from some major editorials state that among the top 30 nations by revenue from digital services as share of GDP more than top 30 nations by revenue from digital services as share of GDP , more than half of them are in the emerging world . In such a scenario, it has emerged that Indonesia is much more advanced that countries such as Sweden and Finland which are Europe's most tech savvy nations

 * India ranks 12th in terms of digital revenues as a share of the economy

 * Since 2017 , digital revenue has grown in emerging countries at an annual pace of 26% compared to 11% in the developed nations

 * Led by e-commerce, the pace of growth in India has been even faster with revenues growing nearly 33% on year over year

* The European Centre for Digital Competitiveness scores G20 nations by the pace of progress in the digital ecosystem and "mindset" and thus puts four emerging nations in the top 5 category - Saudi Arabia , Indonesia , China and Argentina

 * The pressing question that has emerged from all this competitive talk is how have the developing nations adopted to digital technologies faster than the developed nations ? The widely acknowledged answer to such a question has come in the form of a simple answer that is Habit and tenacity. When some are wearing off and losing out on sheen as a result of their comfort seeking, some others rise who have the tenacity to struggle and get more resources for themselves as prize and with analogy it can be only reduced a cyclic process of growth and decay , decay and growth which applies to everyone be it person , society , principles , economies and even countries and continents .. and as well as the planet too (too much philosophy in an nutshell .... )

* The digital divide and access to information and services is narrowing down which used to be the key parameters upon which the developed nations had been brought up and this is trickling down to developing and under-developed nations too .

 * In the last decade alone , the number of internet users have doubled in the G20 nations which is astounding as a decade back access to internet was a distant dream for so many . (I first lay my hands upon internet services in the year 2008 , and the first time I got to own a computer was in the aftermath of 2003 ) . From that to this has been years marked with a boom in technological development which has been significantly very high in the last two decades alone compared to what the last century or any of the centuries ever saw in the history of the world .

 * Robustness of the governments to improve productivity by moving more and more services over to online format in order to make them more transparent and less vulnerable to corruption , which is one of the biggest deterrents / obstacles / vulnerabilities to corruption , which is one of the biggest deterrents / obstacles of doing business in the emerging world , which thereby creates a greater sense of trust and cooperation which in turn facilitates higher economic growth .

 * According to reports , the signle most factor for such kind of marked development to occur , the cost of starting or opening up business is a major driving factor which deters entrepreneurs from taking their feet out to start a business , innovation or self-sustained effort ; but since the cost of such factors has been in the decline in emerging or developing countries coupled with their in government support, guidance, motivation and cooperation such prospective entrepreneurs get a guided push for starting up as they can start their businesses and tools on their own and affordably , which these days could be just monitored and controlled within the grasp of their palms over a smartphone with internet .

 

* But as economies start evolving and growing, chances may arise that the things which catapulted such growth coupled with the rise of globalisation and access to great quality products from around the globe through way-to-go retail apps would slowly and slowly come down again and perhaps this could be a smaller miniscule side-effect of indigenisation which is again a cyclic effect which takes into factor indigenisation and globalisation which would also go all the time (You see ... one has to always keep in mind the cyclic effect of all things to understand all concepts of science , arts , culture , trade , war and even peace ... which encompasses the all round "theory of everything" )

 

 

Tuesday, April 13, 2021

Future Shock of Technology - Cyber Threats on the Rise

 

Future Shock of Technology - Threats on Rise

 

* India is on a high alert after several cyber attacks have jeopardised the operations of major business houses and market establishments in the aftermath of UpStock's data theft

 

* One of the articles in a major newspaper circulated in India states that “With conventional weapons of mass destruction having reached frightening proportions, incentivisation of cyber warfare has become a daily news affair which requires a few resources and could be carried out discretely "

 

* So what is Cyber-Warfare.. " Cyber-Warfare by its very nature is well suited for grey-zone warfare where offensive activities are carried out below the thresholdof all out war and assymmetric attacks " . So all of those who do not have a great idea of this can relate to those scenes in hi-fi sci-fi movies over where Hackers launch their malicious automated code at targeted companies / individuals / institutions in order to either usurp classified data or usurp money in order to jolt the financial system of targeted entity or launch a series of malicious code which would infiltrate into their database and throw the complex database system out of order in order to de-stabilize the system or corrupt the system as these days most of the organisations have their secured systems connected to the world wide web and not all have secure and robust tech to deal with a cyber invasive attack of a gigantic to deal with a cyber invasive attack of a gigantic proportion until and unless the organisations that keep vigil let go off their vigilance and purport themselves as a party involved in it .

 

* It is mentioned and also needs mentioning that these could be classified as a type of threat from both China-Pakistan axis or single entities .

 

* The Union Home Ministry recently informed the Parliament that cyberattacks have risen nearly 300% during the last year amidst the growing Covid Pandemic

 

* Additionally, the Union Power and Resources Ministry of the GoI has admitted that state sponsored Chinese hacker groups have tried targeting India's critical power infrastructure . One such group whichhas surfaced called as "Red Echo" was behind the Mumbai power outage last year

 

* The article has cited its apprehensions and urges the netizens to imagine the chaos that could be caused if by chance a Chinese or Pakistani cyber strike on an Indian nuclear facility happens in future . In such a scenario , the country should cultivate both "defensive and offensive" cyberwarfare capabilities . Chief of Defense Services Mr Bipin Rawat recently revealed that the country is taking steps to counter China's cyber warfare through risk mitigation strategies , building firewalls and recovery systems and integrating the firewalls and recovery systems and integrating the three services cybersecurity resources . But it is touted that India is still way behind China in Cyber Crimes and Offences . And in order to bridge the difference, GoI (Govt of India)has to work in close range with other higher powers like USA and Russia in order to quickly upgrade their cyber tech . The article in its closing points mentions that , this being the need of the hour is a major arena where the Quad nations need to coordinate .


Thursday, April 8, 2021

Testing MapReduce Programs - An introductory Article on Testing of MapReduce Programs for Load and Performance

 



    
    Testing MapReduce Programs

 

* Mapper programs running on a cluster are usually complicated to debug

 

* The best way of debugging MapReduce programs is via usage of print statements over log setions in MapReduce programs

 

* But in a large application where thousands of programs may be running at any single point of time , running the execution jobs and programs over tens or thousands of nodes is preferred to be done in multiple stages

 

* Therefore , the most preferred mode of execution of any program is :

(a) To run the programs using small sample datasets ; this would ensure that what so ever program is running , the program is running in an efficient and robust manner . And for checking of the same , the tried and tested formula for applying the working proficiency of the program over a small dataset is done followed by applying the same over a bigger application / bigger dataset / more

number of testcases etc

 

(b) Expanding the unit tests to cover larger number of datasets and to run the programs over a bigger/larger cluster of network applications . As mentioned in the earlier point , the scope of execution of the testcases is enhanced by application of unit testcases over larger datasets in order to check the robustness

and performance of the system application software

 

(c) Ensuring that the Mapper and the Reducer functions can handle the inputs more efficiently . This means that the set of Mapper and Reducer functions created to work over the split input data would work in cohesion or in tandem with MapReduce programs desired working condition to produce serial output in desired format (text,key-value pair) etc

 

* Running the system application against a full dataset would likely expose more issues , which might lead to rise of undue errors , undiscovered issues , unpredictable results , undue fitting criterias and so on type of issues over the software because of which it might not be that conducive for the system analyst to put the entire full dataset to test over the software . But after all necessary

unit testcases have been checked and working criteria and pre-requisites have been fulfilled , one may put the program to be tested over bigger datasets ; by and by making the work of the MapReduce job easier to run , thereby also enhancing

speed and performance issues gradually

 

* it may be desirable to split the logic into many simpler Mapper and Reducer functions , chaining the programs into single Mapper functions using a facility like ChainMapper library class built within Hadoop (I am yet to explore all the scopes , ChainMapper library class built within Hadoop (I am yet to explore all the scopes ,

specifications and functionalities of the ChainMapper Library which I shall try to cover in a forthcoming session ) . This class can run a chain of Mappers followed by a Reducer function , followed again by a chain of Mapper functions within a single MapReduce Job

 

* More over testing of MapReduce Jobs / Execution of MapReduce Jobs / Analysis and Debugging of MapReduce Jobs would be done in later articles under appropriate headers and titles .

 

Map Reduce Data Types and Formats ( A short explanatory Article )


 

Map Reduce Data Types and Formats


 * MapReduce 's model of data processing which includes the following components : inputs and outputs as the Map Reduce Functions consist of Key-Value pairs


 * The Map and Reduce functions in Hadoop MapReduce have the following general form which is an accepted mode of representation

 

Map : (K1,V1) -> list(K2,V2)

Reduce : (K1,list(V2)) -> list(K2,V3)

 

* The Map input key and value types ( K1,V1 ) are different from the Map output types of (K2,V2)

 

* However , Reduce input takes in K1 and list values of V2 (which is different in format from that of the Map input over Key1 and associated value V1) . And yet again the output for the Reduce process is a list of the key-value pair (K2 and V3) which is again different from that of Reduce Operations .

 

* MapReduce can process many different types of data formats which may range from text file formats to databases .


* An "input split" is a chunk of the input that is processed by a single Map function .

 

* Each Map process processes a single split where each of the split is divided into records and the map function processes each record in the form of a key-value pair

 

* Splits and Records are a part of logical processing of the records and even maps to a full file / part of a file / collection of different files etc

 

* In a database context , a split corresponds to a range of rows from a table / record .

 

MapReduce Programming - An introductory article into the concept of MapReduce Programming

 


MapReduce Programming

 * A Data Processing problem can be transformed into a MapReduce Model by the usage of MapReduce Programming

 * The very first step within the process is to visualize the processing plan of a Map and Reduce Programming problem in a step by step process

 * When a problem involving Map and Reduce Programming gets more complex , the underlying complexity within the Map and Reduce problem can be manifested and resolved in either of the two ways or a combination of two ways

 1) Having more number of MapReduce Jobs -- which would eventually increase the load present over the processors and then mitigated by parallel distribution over the servers

 2) Having more complex Map and Reduce Jobs -- under this scenario one may suppose that the number of sorting jobs and processes might get increased tremendously which might add to the complexity , otherwise complexity might also get enhanced under conditions when more and more key and values for same set of text/words are found out by the program and thus mapping their frequency to the matched key becomes more and more which would again add to the complexity of the Map-Reduce Program . Having more but simple MapReduce jobs leads to more easily maintainable Map and Reduce Programs .

 

Wednesday, March 10, 2021

Using FOR loop with Dictionaries in Python - sample example

 

Using FOR loop with Dictionaries


* FOR loop is very convenient to retrieve the elements of a dictionary .One can consider a simple dictionary that contains color code in smallcase and the corresponding color combination in the following manner and its name as the following :

 

colors = { "r":"Red" , "g":"Green" , "b":"Blue" , "w":"White"}

 

Here , 'r','g','w' characters represent the keys and "Red","Green","White" indicate the values represented by the Keys  Suppose one wants to retrieve only keys from "colors" dictionary , then one can use a for loop in the following manner :

 

for k in colors:

    print(k)

 

In the above loop , one can find that the iterator variable 'k' stores each element of the colors dictionary . Here the iterator , 'k' assumes only the keys and hence the loop displays only keys . Suppose , one wants to retrieve values associated from the keys , then one can obtain them by passing the key to the colors dictionary in the form of colors(k) . The following for loop retrieves all the color values from the colors dictionary .

 

for k in colors:

    print(colors[k])

 

Since these values are associated with keys , one can retrieve them only when we mention the keys . And suppose, one wants to retrieve both the keys and their corresponding values , then one can use the items method within the for loop in the given manner :

 

for k , v in colors.items():

    print(' Key = { } Value = { } '.format(k,v))

 

In the preceding code , the colors.items() method returns an object by the name of 'dict_items' that contains key and value pairs . Each of the value pairs is stored in "k" .

  

==========================================================

Python program to show the usage of for loop to retrieve the elements of dictionaries

# One can use a for loop with dictionaries in the following manner

colors = { 'r' : "Red" ,'g' : 'Green' , 'b':'Blue' , 'w':'white'}

# display only keys

for k in colors:

    print(k)

# pass keys to the dictionary and display the values

for k in colors:

    print(colors[k])

# items method returns the key and value pair into k,v

for k,v in colors.items():

    print('Key={ } Value={ }'.format(k,v))

 

Output

b

wr

g

Blue

White

Red

Green

Key=b Value=Blue

Key=w Value=White

Key=r Value=Red

Key=g Value=Green

 

In the similar manner , another program can be written to count the number of times each letter has occurred in a string .

For example , if "Book" is a string and one is supposed to find the number of occurrences of each letter in the string , then it means that the letter "B" has occurred for 1 time , and the letter "o" has occurred for 2 times and the letter 'k' occurred for 1 time .

In this program , one can use the "get()" method very effectively . Therefore , please recollect that the get() method is used on a dictionary object item to retrieve the value by giving the key .

If the key is not found within the dictionary , then it returns some default value .

The format of the get() method is as follows :

dict.get(x,0)

This statement says that if the key 'x' is found in the dictionary 'dict' , then one must return the 'value' from the dictionary else it would return 0 as result .

One can consider the following code in order to follow or understand the manner in which the combination of key and value for a given string sequence could be mapped to a dictionary and then it could be fetched back from the code :

 

dict = { }

str = "Book"

for x in str:

    dict[x] = dict.get(x,0) + 1

 

In the given code , the last statement should be considerd once again for better understanding and clarity .

 

dict[x] = dict.get(x,0) + 1

 

If one observes the right hand expression with "get()" method then one can observe that it says if 'x' is found in the dictonary 'dict' then it should return a value otherwise it should return a 0 . As we added a '1' to the value returned by the "get()" method and therefore , if 'x' is not found , then it returns a value of 1 .

 

But if 'x' is found , then it returns the value of 'x' plus 1 . One can observe the left side expression , that is dict[x] . Here , this represents 'x' is stored as key in the dictionary . Therefore , whatever the value is returned by the right side expression , the value would be stored into that dictionary as value

for the key 'x' . This means that :

 

dict[x] = value returned by get() + 1

Let's consider the first letter of the string that is "B" . Since the dictionary is initially empty , there are no elements within the dictionary and therefore 'B' is not found within the dictionary .

 

Therefore , dict.get(x,0) + 1 returns the value 1 . In the left side , we have dict[x] which represents dict['B'] . Here , 'B' is taken as the key . And hence the statement becomes :

 

dict['B'] = 1

 

This means 'B' is stored as a key and the value 1 is stored as a value within the dictionary which means 'B' is stored as a key and the value 1 is stored as a value within the dictionary 'dict' . So , the dictionary contains a pair of elements as {'B':1} .In the next step , 'o' is the letter for which the get() method searches in the dictionary .

 

This is not found within the 'dict' and therefore 1 is returned as result . Therefore ,

 

dict['o'] = 1

 

stores the new key and value pair that is 'o' and 1 into the dictionary object 'dict' and hence the dictionary contains the following data items :

{'B':1,'o':1}

In the next repetition of for loop , we get 'o' into the value of 'x' . And since , the value is already available in the dictionary 'dict' , the value that is returned is 1 by usage of the get() method for which the value 1 is added .

 

dict['o'] = 2

 

This means that the old value of 'o' is now updated to 2 in the dictionary and 'dict' contains the elements : { 'B':1,'o':2 } . In this manner , the 'dict' stores each letter as key and its number of occurrences as value .



Ref : Core Python Programming  by Dr.Nageswara Rao (Dreamtech Publications )