This technical blog is my own collection of notes , articles , implementations and interpretation of referred topics in coding, programming, data analytics , data science , data warehousing , Cloud Applications and Artificial Intelligence . Feel free to explore my blog and articles for reference and downloads . Do subscribe , like , share and comment ---- Vivek Dash
Saturday, May 1, 2021
Saturday, April 24, 2021
Friday, April 23, 2021
Describing the use of Statistics in Machine Learning - A full detailed article on some of the most important concepts in Statistics
Describing the use of Statistics
* Its important to skim through some of the basic statistical concepts related to probability and statistics . Along with that , we will also try to understand how these concepts can help someone to describe the information used by machine learning algorithms
* The zist of all the learning of these concepts is not only
about how to describe an event by counting the number of occurrences , but its
about describing an event without counting every time how many times a
particular event occurs .
* If there are some imprecision in a recording instrument that one uses , or simply because some error in the the recording procedure of a machine occurs , rather an imprecision occurs in the instrument that one uses or simply because of any random nuisance which disturbs the process of recording a given measure during the process of recording the measure occurs ... then a simple measure such as weight , will differ every time one would get a scale which would be slightly oscillating around the true weights and minimal variation scale . Now , if someone wants to perform such a small incident in a city and want to measure the weight of all the people in the city , then it is probably an impossible experiment to be conducted on such a large scale as it would involve taking the weight-wise reading of all the people in the city which is something that is practically not possible , because first of all if someone wants to perform this experiment in one go , then one has to create a big big gigantic weighing scale to mount all the people of the city in its weighing pans , which is completely an impossible task , and probably the scale may break once all the people have been mounted to the pan or otherwise the worst thing that may happen is that once all the people's weights have been measured , the experiment could render itself insignificant as the experiment once conducted would make the use of the weighing machine useless and hence the cost associated with building of such a big machine for carrying out just one task would become meaningless .
* So , the purpose of experiment might get achieved , but the cost of the built-up of the instrument would run so high that a big dent in the overall GDP of the city would get created which might cripple the city's finance and budget . On the other hand , if we take the measurement of the entire city's weight recording each person's weight one by one , then the effort and time taken for the entire activity to be completed might take some weeks or months of time . Because of the high amount of time and effort that would get consumed while managing the entire ruckus won’t suit the idea for adaptability and taking up the idea. And even if all the weights of the people residing in the city is successfully measured, there are a lot of chances that anyhow some amount for error would definitely popup making the idea of the entire process not so fruitful and fault-proof
* Having partial information is a quite complex process which is not a completely negative condition because one can use such smaller matrices for the purpose of efficient and less cumbersome calculations. Also on top of that, it is said that one cannot get a sample of what one wants to describe and learn because the event's complexity might be quite high and may probably feature a great variety of features . Another example that some users could consider while taking a case of a sample or a large case of data , is a case of Twitter tweets . All the tweets may be considered as some sample of data over where the same data could be treated as some experimental potions and minerals which are processed using several word processors , sentiment analyzers , business enhancers , spam , abusive data and all depending upon the sample of data associated with each of the text within the short frame of data that one can provide within the text section .
* Therefore it is also a good practice in sampling to sample similar data which has associated characteristics and features which will present the sample data in the form of a grouped cohesive data which fit into a proper sampling criteria. And when sampling is done carefully, one can get an idea that one can obtain a better global view of the data from the constituent samples
* In Statistics , a population refers to all the events and objects that one wants to measure and is a part of the given criteria which gives in detail the account of metrices of the population . Using the concept of random sampling , which is picking the events or objects one needs to choose one's examples according to the criteria which would determine how the data is collected ,assembled and synthesised. This is then used for feeding into machine learning algorithms which apply their inherent functions for determination of patterns and behaviour.
* Along with such determination , a probabilistic model of input data is built which is used for prediction of similar patterns from any newly input data or datasets ,Application of this concept of data generation from population's subsamples and mapping the identified patterns to map new use cases is one of subsamples the chief objectives of machine learning on the back of supported algorithms
* "Random Sampling" -- It is not the only approach for any sort of sampling . One can also apply an approach of "stratified sampling" through which one can control some aspects of the random sample in order to avoid picking too many or too few events of a certain kind .After all , it is said that a random sample is a random sample , the manner it gets picked is irrespective of the manner in which all samples would criterion themselves for picking up a sample , and there is no absolute assurance of always replicating an exact distribution of a population .
* A distribution is a statistical formulation which describes how to observe any event or a measure by ideating the probability of witnessing a certain value . Distributions are described in Mathematical formula and can be graphically described using charts such as histograms or distribution plots . The information that one wants to put over the matrix has a distribution , and one may find that the distributions of different features are related to each other . A normal distribution naturally implies variation and when dealing with numeric values , it is very important to figure out a center of variation which is essentially a value which corresponds to the statistical mean which can be calculated by summing all the values and then dividing the sum by the total number of values considered .
* Mean - This is specifically a descriptive measure which tells the users the values to expect the most from within dataset . as it is a general fact that most of the times , one can observe that the mean of a dataset is that data which generally hovers around a given data group or the entire dataset . The Mean of a dataset is the best suited data for any symmetrical and bell-shaped distribution . In cases , when the value is above the mean of the entire dataset , the distribution is similarly shaped for the values that lie below the mean . The normal distribution or the Gaussian distribution is shaped around the mean which one can find only when one is dealing with legible data which is not much skewed in any direction from the equally shaped domes of the normal distribution curve . In the real world , in most of the datasets one can find many skewed distributions that have extreme values on one side of the distribution , which influences the value of mean so much
* Median - The Median is a measure that takes the value in the
middle after one orders all the observations from smallest to the largest
values within the dataset . Based on the value order, the median is a less
approximate measure of central approximation of data .
* Variance - The significance of mean and median data descriptors is that they describe a value within a data description around which there is some form of variation . In general, the significance of the mean and median descriptors is variation. In general , the significance of the mean and median descriptors is that they describe a value within the distribution around which there is a variation and machine learning algorithms generally do not care about such a form of variation . Most people generally refer to the term , variation as "variance" . And since , variance is a squared number there is also a root equivalent which is termed as "Standard Deviation" . Machine Learning takes into account the concept of variance in every single variable (univariate distributions) and in all the features together (multivariate distribution) to determine how such a variation impacts the response obtained .
* Statistics is an important matter in machine learning because it conveys the idea that features have a distribution pattern . Distribution of data implies variation and variation means quantification of information ... which means that more amount of variance is present in the features , then the more amount of Information can be matched to the response .
* One can use statistics to assess the quality of the feature
matrix and then leverage statistical measures in order to draw a rule from the
types of information to their purposes that they cater to .
Thursday, April 15, 2021
Tech Commandments for a safer digital life - 5 principles to adhere to when using Internet
Tech Commandments for a safer digital life
* Technology has become a mammoth sized factor in our daily
lives and to top it all , Technology is always on the change which means that
one should always stay vigil of the changes happening around us as many
perpetrators and miscreants are on the lookout for finding new ways to
infiltrate into the loopholes that we set knowingly or unknowingly which leads
them directly in hold of sensitive information which can lead to a big
devastation if left unguarded and unprotected
* Some experts from leading security firms assert that one
should always remember that any piece of our identity that we post online could
eventually be used by fraudsters and hijackers / hackers to pervade into our
online accounts for which one needs to keep oneself and connected members safe
when online . These days bots ( trackers) can track any account and collect account
information at any point of time
* Therefore in order to keep oneself on the guard and stay safe
, some of the commandments that one needs to adhere to all the time are the
following :
1) One should not overshare personal info
As many of us use high ended camera phones with very high or unlimited
storage capacity , one gets into a clicking mode and becomes a self-proclaimed cameraman
/ photographer . But these days thanks to highly developed AI programs which
can take even pictures as input parameters and retrieve all relevant
information from the photo , one should stay vigil about the photos that one
clicks , location , people in that photo , context and backdrop for that photo and
several other factors before one gets into clicking mode to showcase one's photography
skills
2) One should not use Weak and Easy to crack
passwords
We all have a tendency to associate easy passwords which are
small and easy to remember for all of our accounts as we normally want to skip
ahead of all the mental work of recalling big and complex things whenever we
want to get into our accounts .. be it social media accounts , bank accounts ,
insurance accounts
(and Swiss Bank Deposit accounts too .. ) . As mentioned , these
days tracker bots can track all the gateways of access that an individual
leaves upon their online pathways , so unrecognised and unregulated access into
our trails can beonline pathways , so unrecognised and unregulated access into
our trails can be
discovered by bots and provide the information to the collecting
agency which is on the lookout of such paths which could be exploited . Many
such agencies pass on these trail paths to hackers to peek into and steal out
money / documents .. anything precious to them (remember .. "precious" of LOTR , one can turn into
Gollum for such "precious" things ) . Therefore , always make use of stronger and lengthier
passwords which is one's somewhat safety check for stopping unwanted and
malicious intrusion . Many people make use of password managers who make use of
multiple accounts , but this is also vulnerable as these are stored in the form
of either xml , json objects in the form of cookies which could be again
collected from the browser plugins that one uses on a regular basis . Thats why
, one should also try disconnecting from cookie storage from browser when one
closes the session . And thus , the best thing to do is to note down all the
important passwords over a piece of paper and keep them at a safe storage space
.
3) One should use Multi-factor authentication or
two-step verification
These days password comprehension can be done is so different
forms by hackers that , if someone wants to any way hack into some account ,
then they will eventually get into the account and that too using several tools
. Thats why most of the security experts recommend that one should make use of
multifactor authentication (two-factor , three-factor etc) in order to access a
given account which involves a user's verification before logging into account
using a system of OTP over phone and authenticator apps that send temporary
always changing codes that ensure that the user who is using the account is the
real one and not a dummy or someone else
4) One should not share data about friends and
Contacts
This comes as a completely new method of data siphoning which
occurs when someone accepts any permissions to any app or software over the
phone . This makes the app-owner a party to the shared information as requested
by the app within the permission page of the application . Therefore, one thing
that one needs to keep in mind is to keep a check over the permission page of
the
applications that one installs .Best is .. one should try to
limitise one's wants , keep few applications and software over one's phone and
do not install those software that require a lot of permissions to be accepted
before making use of the software .
5) One should always stay vigilant and skeptical
These days all the security experts accept the one rule of thumb
for all security practices -- "
Trust No One in this Greed Infested World" . Whenever you recieve any
call ,message , email soliciting any personal information .. then do not trust
any of message , email soliciting any personal information .. then do not trust
any of the mails . This could be a phishing attack from someone who wants to
profit out of undoubting people who out of trust and foolishness get entrapped
in their untrustworthy traps , thereby losing security over their devices and
mediums .Fraudsters nowadays can embed malware over legitimate looking emails
within hyperlinks which once clicked could install unsuspecting software over
your system all without anyone's coming to know of such a background activity
running on your system . Therefore, whenever any suspicion occurs , always opt
out of such apps , softwares , emails , anything (sigh ... wish I had known
these earlier )
These days I personally feel .. the days of Nokia 1100 and Nokia
1600 , movies without real-looking CGI , games like Tetris and Mario were the
best .
Fast Spreading Digital Adaptation in Emerging Nations and the associated "Theory of Everything" in the trade and commerce world - An article by Vivek Dash
Fast Spreading Digital Adaptation in Emerging Nations and the
theory of Everything
* Emerging economies have been struggling with growth through the 2010's and still the feeling of pessimism clouds over the present decade whether the steadfast growth trend could be seen to be going strong or whether the trend would be gauged down as it is being seen that People have accrued high amounts of debt during the ongoing pandemic which has become a detrimental factor to growth in not only emerging countries but also countries which are developed like USA , Russia , China , Japan , Germany , France , UK , Finland as questions over Growth tapping and keeping the same pace steady even during a pandemic period is of paramount importance to all
* As per some key metrices and data collected by popular
editorials , the onslaught and adaptation of digital revolution is much more in
emerging economies than that of developed nations and has been growing at a
much higher rate year on year .
* The European Centre for Digital Competitiveness scores G20 nations by the pace of progress in the digital ecosystem and "mindset" and thus puts four emerging nations in the top 5 category - Saudi Arabia , Indonesia , China and Argentina
* The digital divide and access to information and services is
narrowing down which used to be the key parameters upon which the developed
nations had been brought up and this is trickling down to developing and
under-developed nations too .
* But as economies start evolving and growing, chances may arise that the things which catapulted such growth coupled with the rise of globalisation and access to great quality products from around the globe through way-to-go retail apps would slowly and slowly come down again and perhaps this could be a smaller miniscule side-effect of indigenisation which is again a cyclic effect which takes into factor indigenisation and globalisation which would also go all the time (You see ... one has to always keep in mind the cyclic effect of all things to understand all concepts of science , arts , culture , trade , war and even peace ... which encompasses the all round "theory of everything" )
Wednesday, April 14, 2021
Noticeable debacles on the part of public and political parties at the outset of Phase-2 of Covid pandemic in India ( a collective analysis )
Covid Phase-2 noticeable debacles
* Central Govt's decision to clear approvals on fast track basis
for vaccines used in other countries is a welcome gesture in recognition of the
sheer magnitude of the surging magnitude of the second Covid wave .
* It is also a grave issue for the officials to adhere to strict
appropriate norms which needs to be followed both on the part of the officials
as well as the public to adhere to best possible appropriate norms along with
spread of the dangerous infection across the country
* Along with the Kumbh Mela of some sort , the Covid Tika Divas
ensues after a grand function in the form of the "Paschim Bengal"
elections which is again a grand amalgamation of people and opinions contained
within a densely packed state population which is widely known for its
celebrations be it religious ,spiritual , political or opinion based , the
state celebrates all the forms of functions on a wide scale with mass
gatherings and rallies of swarming people .
* This should have atleast called for sensitizing the general populace to adhere to all the mandatory guidelines , code of conduct for assemby and get togethers or rallies but it seems all these have again went for a toss as masses were noticed with no masks . Same situation is again on the notice in neighbouring Odisha where Panchayat Elections are to be held . Again precautions need to be set in place for people who attend any form of rallies or public gatherings as this is the utmost and basic need of the hour in order to curb the spread of the virus which travels in air and gets propagated from infected (be it mild/moderate/severe case) person to person
* I am no authority or person in charge to look upon the
persisting conditions and what should someone be mindful about when people are
in public during these pandemic periods of disease transmission but still then
I think some there should be enforcement of law and order by people in charge
of high offices within the confines of govenmental powers and positions to keep
inculcating civic sense through all possible mediums of communication be in
print or through telecom to convey the guidelines at all possible times as I
and probably most of the people less or more similar to me have shorter memory
spans and attention deficit disorder because of which getting misdirected is
not so uncommon , but as because times are critical and the forecast for the
second phase of pandemic
is an onset of a ghastlier trail of infections and deaths , a
diligent adherence of strictness in the coming months is of paramount
importance
* The current visuals from Gujarat , MP and Chattisgarh is not much encouraging and looks like the second wave has hit hard which is a grave warning to rest of the people in other states to adhere to proper Covid behaviour , mask up rather double mask up and to get vaccinated ... ( though I am yet to get vaccinated or receive vaccine, as I am not yet sure which vaccine should one take ... out of all the existing known options .. lol )
* Another funny thing I came across in one article is how some big politicians and Netas have started demanding VIP treatment from medicals and hospitals in the wake of this crisis which has infuriated some of the doctors and doctors associations.. Here the doctors forget to just chillax as we have been taught to idolise our great politicians , cricketers and filmstars as nothing less than demigods , and so getting a grand treatment from public including doctors , engineers is a sort of entitlement which nobody can negate .. so chillax .
* Another point that I would probably like to bring into picture
is approval of foreign vaccines without passing through phase2 , phase-3 or whatsoever
number of trials to be catered to India's population is a digrace from natural
biological behaviour as what I presume is that the vaccines which have been
procured from other countries may not be that much effective in our case as
most of the people of India and south-east Asia have a different genetic makeup
which is different from that of Americas , or that of Europe , Australia and
hence administering the vaccine which has been created from samples belonging
to other countries may not work out well ( Though I have to be optimistic about
them as my domain and forte are different , hence I may not call myself an
analyst in this field )
* I would acknowledge once again that some of the things that I
have mentioned in this collected and referred article is an opinion of my own
and may be seen as a case of counter opinion to popular opinion from a general
public ( You can happily call me a dimwitand nota virtuoso if some of the
points have not been put very appropriately as written by noted columnists and
writers for lack of great jargons and plausible points or the lack of proper
statistical figures ... its all upto the readers discretion )
Govt fast-tracks nod for vaccine procurement from other countries on emergency basis
Govt fast-tracks nod for vaccine
* Vaccines authorised by World health Organisation or Vaccine
Regulators in US ,Europe , UK and Japan would be granted an emergency use
approval in India mandating requirement of post-approval parallel bridging
clinical trials
* First 100 beneficiaries of such vaccines shall be kept for
assessment for 7 daysfor safety outcomes before they are rolled out across the
entire country as per reports in the leading newspapers in India
* Among the healthcare giants , one of the largest drug
manufacturing organisation Pfizer has a production capacity of 2.5 billion
doses per annum and almost 1.6 billion confirmed dose purchases . From this one
can confer that India has 900 million doses that India could tap into for
procurement.
* Moderna and Johnson&Johnson
, two of the largest drug manufacturers and retailers have also bid in equal
terms for the same and in the process have taken in more orders than what could
be produced
* On 12th April , Govt of India also granted permission
permission for restricted emergency use of Russian vaccine Sputnik V
* The aforementioned regulatory approvals are intended to
increase the availability of the jabs amidst a steep second wave of the
infection .
* The vaccines can be imported in a ready- to use vial cylinder
bottles or in the form of fill and use form
Tuesday, April 13, 2021
Russian made covid-19 vaccine "Sputnik-V" to be used for emergency use in India
Russian made covid-19
vaccine "Sputnik-V" to be used for emergency use in India
* Expert panel on drugs and vaccine regulation and adminstration
in India has cleared "Sputnik
V", Russia's largely used vaccine for circulation within India
* The administration has given in to this proposition amidst
high concerns over the need to step up Covid-19 vaccine and dosages manufacture
and supply within India as there is a wide bridge in demand and supply within
India and thus the need for procuring the vaccine for restricted emergency use
within India .
* The final approval upon this is to be taken by the Drug
Controller of India very soon as per reports over some of the major daily
newspapers circulated in India
* Once this proposal would be approved , Hyderabad based drug
manufacturer company "Dr Reddy's Lab" will initially procure the
importing license post which it will import from Russian manufacturing company
and then it will supply to GOI
( Government of India )
* The GOI will procure "Sputnik V" from Dr Reddy's Lab
for the national Covid vaccination programme but this will be in small
quantities in the initial period.
* Stocks of the same are expected to rise by June-July 2021 as
per published reports in the newspapers
* The Central govt expects that "Sputnik V" could be
seen as a major welcome addition to "Covishield"
and "Covaxin" which
are India's inhouse conceptualised, developed , manufactured and
served vaccine for the citizens as well as for export to interested countries .
This is expected to improve the lurking gap between the prevalent supply and
demand gap as per the current trend
* It is however expected that with increased production of Covaxin , India's dependency over other countries for vaccine will fall and India might be able to cater to its citizens needs and welfare as per the dictats of AtmaNirbhar Bharat
Double Shield - Two Mask Theory ( Prevention is Better than Cure - Strategy for Covid-19 new impacts )
Double
Shield - Two Mask Theory
* As the second wave of Corona Virus Pandemic hit the country ,
many of the people have accepted a double-mask approach in order to keep the
corona virus at bay as experts said that this is one of the most "advisable"
and "best" methods to stop transmission
* Most of the people who have started practicing this trend have started wearing a surgical mask and a cloth mask or two cloth masks .
* Also experts say that as many of the common masks that we generally use do not fit well and tightly to our faces , wearing a double mask definitely reduces the risk of the droplets from any infected person susceptible to escape from the sides of the mask either while breathing in/out or sneezing etc
* The double mask recommendation is based on a study conducted by US Centre for Disease Control and Prevention (CDC)
* The CDC has conducted experiments to assess two ways of
improving the fit of the medical procedure masks - ie fitting a cloth mask over
a medical procedure mask and knotting the ear loops of a medical procedure mask
and then tucking in and flattening the extra material close to the face which substantially
improved source control and reduced wearer exposure .
Future Shock of Technology - Cyber Threats on the Rise
Future Shock of Technology - Threats on Rise
* India is on a high alert after
several cyber attacks have jeopardised the operations of major business houses
and market establishments in the aftermath of UpStock's data theft
* One of the articles in a major newspaper circulated in India states that “With conventional weapons of mass destruction having reached frightening proportions, incentivisation of cyber warfare has become a daily news affair which requires a few resources and could be carried out discretely "
* So what is Cyber-Warfare.. " Cyber-Warfare by its very nature is well suited for grey-zone warfare where offensive activities are carried out below the thresholdof all out war and assymmetric attacks " . So all of those who do not have a great idea of this can relate to those scenes in hi-fi sci-fi movies over where Hackers launch their malicious automated code at targeted companies / individuals / institutions in order to either usurp classified data or usurp money in order to jolt the financial system of targeted entity or launch a series of malicious code which would infiltrate into their database and throw the complex database system out of order in order to de-stabilize the system or corrupt the system as these days most of the organisations have their secured systems connected to the world wide web and not all have secure and robust tech to deal with a cyber invasive attack of a gigantic to deal with a cyber invasive attack of a gigantic proportion until and unless the organisations that keep vigil let go off their vigilance and purport themselves as a party involved in it .
* It is mentioned and also needs mentioning that these could be classified as a type of threat from both China-Pakistan axis or single entities .
* The Union Home Ministry recently informed the Parliament that cyberattacks have risen nearly 300% during the last year amidst the growing Covid Pandemic
* Additionally, the Union Power and Resources Ministry of the GoI has admitted that state sponsored Chinese hacker groups have tried targeting India's critical power infrastructure . One such group whichhas surfaced called as "Red Echo" was behind the Mumbai power outage last year
* The article has cited its apprehensions and urges the netizens to imagine the chaos that could be caused if by chance a Chinese or Pakistani cyber strike on an Indian nuclear facility happens in future . In such a scenario , the country should cultivate both "defensive and offensive" cyberwarfare capabilities . Chief of Defense Services Mr Bipin Rawat recently revealed that the country is taking steps to counter China's cyber warfare through risk mitigation strategies , building firewalls and recovery systems and integrating the firewalls and recovery systems and integrating the three services cybersecurity resources . But it is touted that India is still way behind China in Cyber Crimes and Offences . And in order to bridge the difference, GoI (Govt of India)has to work in close range with other higher powers like USA and Russia in order to quickly upgrade their cyber tech . The article in its closing points mentions that , this being the need of the hour is a major arena where the Quad nations need to coordinate .
Monday, April 12, 2021
Odisha : Travel details mandatory for air and train passengers without -ve reports
Odisha : Travel details mandatory for air and
train passengers without -ve reports
* All Train and Air passengers without RT-PCR negative reports
or vaccination reports or vaccination certificates would have to furnish their
details at Bhubaneswar's Biju Patnaik International Airport and the city
railway station , as per orders released from Monday .
* Following this , all the passengers have to register
themselves following which exit passes would be issued to the registered
passengers
* According to the orders and press release of the state
government , the Bhubaneswar Municipal Corporation (BMC) will scrap the
practice of spot RT-PCR testing
* Currently , any passenger arriving at the railway station has
to produce an RTPCR negative report or a proof of the full dose of vaccine
taken as a part of standard operating procedures released as standard protocol
for allowing transit for the arriving passengers
* In case , a passenger fails to produce any of the above
details , the passenger would have to go through the registration counter ,
fill in a form furnishing the details of the place .. they are from and the
designated destination they are up to
* After the final registration process is over , the passenger
would be provided with an exit pass which he/she would have to show at the
place of exit where the Railway Police would be deployed to check
* For passengers belonging to Bhubaneswar , the BMC would be
keeping their data for further tracking , monitoring and follow-up processes
* This data of the number of outbound passengers ( going to
other different districts ) would be sent to the respective district
adminsitrations over email
* The current footfall numbers recorded for Bhubaneswar is around 20,000 passengers per day with over 60 trains passing through Bhubaneswar on a daily basis