Data Science Capstone project - The battle of neighborhoods in Dubai

1. Introduction/ Business understanding

1.1 Description of problem

Recently, the 13^th edition of IPL (Indian Premier League) has been announced amid coronavirus pandemic and UAE has been chosen as the host country. The league is slated to commence from 19^th of September, 2020. There has been ongoing discussion regarding the entry of audience in the stadium. The dataset of Dubai has been used to help the visitors of Dubai find places suitable for restaurant, hotel and so on during the IPL season.

1.2 Background of problem

Indian Premier League (IPL) is one of the most popular and highly valued league across the world particularly within cricket playing nation like India, Australia, England and so on. It is an India’s version of T20 cricket league tournament. It gathers large audience in stadium and has huge viewership across cable TV and digital platform. Since, it is India’s tournament it is mostly played in India. But in some extra ordinary condition, it is played in some other countries. This time it is UAE. UAE is also a cricket playing nation and has similar time zone as of India. Many games are slotted to be played in the stadium of Dubai as well. Dubai is located in the eastern part of Arabian Peninsula on the coast of Persian Gulf. Dubai aims to be the business hub of western Asia. It is also a major global transport hub for passengers and cargo.

It is difficult for new travelers to find best place suited for them. So, using the foursquare API, I have performed various analysis on the data set of Dubai to find the best place for restaurant, hotels, parks and so on. This could help the new visitors of Dubai to get the overview of the places.

2. Data description

The dataset that used in this project is of Dubai scrapped from Wikipedia. This dataset contains the list of 131 communities of Dubai.

Data source:

https://en.wikipedia.org/wiki/List_of_communities_in_Dubai

We scrapped the data from the table of Wikipedia using a python library called ‘Beautiful soup’. We will use only 3 columns of the dataset i.e. Community Number, Community (English) and Community (Arabic).

Example of dataset:

I used ‘geopy’ library to find the latitude and longitude of each community. And then using foursquare API I found the venues in each community and what is each community famous for.

3. Methodology

3.1 Scrapping table of list of communities of Dubai from Wikipedia

I first read the table of Wikipedia and then iterating through data of each rows, a new data frame was created with 6 columns.

 Only 3 columns were kept and the rest were dropped. The columns were renamed. The new data frame looked in this way:

3.2 Adding geospatial data

Using geopy library location i.e. latitude and longitude of each communities were retrieved. The communities whose location could not be found were left out. Hence, I was left with 65 communities out of 131.

The outlook of data after adding location:

3.3 Finding the venues of neighborhood within a radius of 500 meters using Foursquare API:

Defining the credentials to connect to Foursquare API:

 First exploring the venues of neighborhood ‘Abu Hail’:

we can see a total of four venues were returned by foursquare.

Exploring the venues of all 65 neighborhood of Dubai:

I was having problem when trying to explore the venues of all 65 neighborhoods at one go. So, what I did was I divided the 65 neighborhoods into 4 groups and then explored the venues of each neighborhood separately. When the venues of all four groups were returned, they were concatenated together.

The data frame after concatenation along with venues of each neighborhood is as follow:

The number of venues in each neighborhood returned by foursquare can be viewed as:

 

From above table we can see Abu Hail has 4 venues, Al Baraha, Al Buteen, Al Garhoud has 40 venues and so on.

Analyzing each neighborhood using one hot encoding:

Displaying each neighborhood with top 5 most common venues:

 

From above figure, we can see the top 5 venues of Al Baraha are Hotel, Middle Eastern Restaurant, cafe, American Restaurant and Spa. The frequency above represents that among 100% venues in Al Baraha, 20% are Hotel, 20% are Middle Eastern Restaurant, 10% are Café, 10% are American restaurant, 10% are Spa and the remaining 30% are venues other than these.

The top 10 venues of each neighborhood are displayed in below table:

Clustering the neighborhoods i.e. communities of Dubai based on the similarities of their venues using K- Means algorithm:

The neighborhoods have been grouped into five clusters.

The K-Means label for each neighborhood:

Now, plotting each neighborhood into map using folium library:

Folium is an essential library to visualize locations on a map. It also allows to zoom in and zoom out the map. With very lines of code, it, does amazing piece of work for visualization of data.

 

 

 4. Results/ Discussions

From the study of venues of each neighborhood we got some results. Lets discuss those results here:

Finding 1:

As we are discussing about IPL going to be held in UAE and being an Indian league tournament more Indians are expected to visit this place. From above data we see plenty of Indian restaurant available here. So, people from India will probably face no problem finding the restaurant of their kind. Also, cricket is mainly considered an Asian game. So, for people from across the Asia visiting the place can also find plenty of Asian restaurant.

The places where one can find Indian restaurant easily are:

 

From above data, we can see Emirates Hill Third, Marsa Dubai, Al Raffa, Al Karama are more famous for Indian restaurant.

Finding 2:

Places where hotel can be found easily are:

So, if someone in Dubai is looking for place with more options available for hotel, they can choose from the places above.

 

Finding 3:

Places with most parks are given below:

So, people fond of parks can choose to stay in the communities/ neighborhoods mentioned above.

Finding 4:

Someone fond of beach can choose to stay in the given below:

Finding 5:

Many people love to have coffee frequently and it becomes for them when they don’t find a coffee shop easily. So, here are the list of places more famous for having coffee shops.

So, these were some findings which I felt were more necessary to be known to people traveling to Dubai.

5. Conclusion

In today's time of digital world, data science plays vital role. It increases the capability of the businesses, medical instruments. It helps the businesses to analyze the behavior of their customers, and also compete with their counterpart in a fast-changing world. With an exponential increase in the use of digital instruments in various sectors, lots of data are being generated and stored every day.  Hence, it becomes quite instrumental and essential to analyze those data to gain information which could help in the improvement of various sectors by taking right decision at right time.

With this project I have made an effort to help the first time travelers to Dubai especially during the season of IPL. I have used some common libraries like geopy, folium to find the location and plot those locations on map respectively. Also, I have made use of foursquare API to explore the venues of each neighborhoods. Despite all these efforts, there are still some areas of improvements which could help in providing even more essential and realistic information from the data.

 

Link to Github

Read More

How to open cmd and powershell in folder in windows

Read More

Create virtual environment in python and install django

Read More

How to uninstall jupyter notebook?

Read More

Install jupyter notebook in windows both with and without virtual enviro...

Read More

Install python package using jupyter notebook

Read More

Open a file in jupyter notebook

Read More

How to permanently disable and clear windows 10 activity history?

Read More

How to delete all the data collected by google/youtube ?

Read More

Solved: Fatal error: Uncaught Error: Call to undefined function mysqli_connect()

Read More

Cricket database project (with CRUD operations) developed using php and MYSQL database

Read More

How to clear the search history of file explorer?

Read More

100% working: How to recover the permanently deleted files for free?

Read More

How to get detailed battery report in windows 10?

Read More

How to add idm extension to tor browser?

Read More

10 best websites to get free online courses from udemy, lynda, bitdegree and so on

1.

https://tutsgalaxy.net/category/tutorials/

2.

 https://tricksinfo.net/

3.

https://www.learningcrux.com

4.

 https://freecourselab.com


5.

https://courseforfree.com

6.

 https://freetutorialsus.org

7.

https://courseclub.me

8.

https://coursecatalog.us


9.

https://freecoursesite.us

10.

 https://coursedrive.org

Read More

Jobs portal web application using django framework

Read More

Solved! search.yahoo.com browser hijacker

Read More

Summary on “An integrated cost model for software reuse”

Today, complex, high quality computer based-systems must be built in a very short time period. This results in an organized approach to reuse. Sometime the potential gains from reusing specifications may be greater than reusing code component because code contains low level details. There are various features distinguishing between different cost models. Investment cycle which has four distinct cycles as corporate, domain engineering, application engineering and component engineering investment cycle. Economic function having five different functions as Net Present Value, Payback Value, Average Return on Book Value, Internal Rate of Return, Profitability Index. Cost factors which specifies what aspects of reuse decision we want to consider. Reuse organization- the organizational structure has some impact on how costs are determined, charged, and accounted for. Scope where some models consider a short-term decision whereas others consider a long-term investment cycle. Some cost models neglect integration costs. Some cost models fail to take into account the discount rate of resources and so on.  There are variety of viewpoints involved in software reuse initiatives like corporate executives, the producer staff, the consumer staff, library managers, and component providers. A generic software reuse model can be classified as Variety of investment cycles, Variety of cost factors, Variety of economic functions, Variety of viewpoints, Variety of hypotheses. Variety of investment cycles involve four different cycles as Corporate level decides Whether to introduce reuse in the practice of software development, Domain engineering decides whether to initiate a domain analysis/domain engineering initiative, Application engineering decides whether to introduce reuse practices for development project, Component engineering decides whether it is worthwhile to develop a specific component to serve a group of project teams. Variety of cost factors include investment Cycle, denoted by Y, measured in number of years, typically ranging between 3 and 5, discount Rate, denoted by d, is an abstract quantity, that typically ranges between 0.10 and 0.20. It reflects the time value of money, investment Costs, denoted by IC, and measured in person months because most costs that arise can best be quantified as personnel effort, episodic Benefits, at year y, for 1 <= y <= Y , denoted by B(y), and measured in person months, episodic Costs, at year y, for 1 <= y <= Y , denoted by C(y), and measured in person months. Variety of economic function include Net Present Value, denoted by N P V, measured in person months, Return on Investment, denoted by ROI. ROI recognizes that investments involve risks, Profitability Index, denoted by PI. This quantity allocates the potential profit with respect to the investment cost. An investment is worthwhile whenever P I exceed 1. Internal Rate of Return, denoted by IRR, and defined as the value of d that makes the net present value zero. Payback Value, denoted by P B and defined as the shortest investment cycle that makes the net present value non-negative. Average Rate of Return, denoted by ARR. It prorates the profitability index by the number of years in the investment cycle. Variety of viewpoint includes Component Engineering Viewpoint where We are deal with here is whether or not to develop a reusable asset to satisfy a tentatively specified set of requirements. Application Engineering Viewpoint where we deal whether or not to adopt reuse in a given development project. Domain Engineering Viewpoint where we deal whether or not to initiate a domain engineering effort in a tentatively specified application domain. Corporate Viewpoint where the investment decision we deal with is whether or not to initiate a corporate software reuse program. Variety of hypotheses includes Non-Linear Cost Effects, Integration Costs, Quantifying Quality Gains, Code Inflation. In automated support, the proposed has two main functions. In archival function, the purpose of this function is to keep track of costs and benefits as they arise. In analytical function, the purpose of this function is analyze investment cycles by producing any combinations of functions. There are various existing cost models. The model proposed by Ganey and Cruickshank combines domain engineering costs and application engineering costs in a single equation, it does not take into account considerations and assumes that the number of applications that make up the domain engineering effort is predetermined. Kain proposes a return on investment model that is especially geared towards ob ject oriented programming (where reusable assets are objects at various levels of abstraction). There are various other models.

Read More

Summary of NO silver bullet

There are two types of complexity in software development process. They are essence and accidents complexity. Essence complexity are inherent in the nature of the software whereas accident complexity is such, that today attend its production but that are not inherent. Accidental complexity arises due to hardware constraints, awkward programming languages. Most of the programmers are still devoted towards solving accidental problem but now the time has come to address essential complexity. Essential complexity include conformity, changeability, flexibility, invisibility. In conformity, the challenge is to run new software on a system where other software systems were running. Changeability makes software dependent on the technological advancement of its surroundings. Because software is invisible, it becomes very difficult to sketch the clear idea of how the final product looks or perform. The development of high level languages like Fortran, C, C++ have solved the accidental difficulties to a great extent. There is no one silver bullet but a series of innovations could lead to a significant improvement. Ada language, object-oriented programming, artificial intelligence could play vital role to achieve those improvements and are hopes for the silver. Building expert systems is not an easy task but systems like graphical programming, program verification, automatic programming play vital role in software development. As large number of software engineers are involved in software development, large number is developed daily. It leads to an attack on conceptual essence and sometime it is more reliable to buy the already built software rather than developing the new one. There is difference between good designers and great designers. There is as much as tenfold difference between an ordinary designer and great one so they should be treated well and provided with higher status. 

Read More

elasticsearch - Could not find any executable java binary. Please install java in your PATH or set JAVA_HOME.

First of all make sure java jdk is installed on your system. Then go to environment variables of your system. Add new variable called "JAVA_HOME" and add variable value as the path where jdk is installed on your system (eg: C:\Program Files\Java\jdk-12.0.1) and then press ok as shown in figure below:

Now, open cmd type echo %JAVA_HOME% ,if it shows the path entered previously while creating new variable then its ok. If everything is all right upto this point, go to the folder when the files of elasticsearch is located through cmd (for eg have my folder on G: drive so my path looks like 
G:\elasticsearch\bin> ). Finally, enter elasticsearch-service.bat install.
If you see output as shown below then your installation has been successful.

G:\elasticsearch\bin>elasticsearch-service.bat install
Installing service      :  "elasticsearch-service-x64"
Using JAVA_HOME (64-bit):  "C:\Program Files\Java\jdk-12.0.1"
The service 'elasticsearch-service-x64' has been installed. 

To run elasticsearch service, again enter given command entering into the bin of your elasticsearch folder as shown below:


G:\elasticsearch\bin>elasticsearch-service.bat manager

Running this command will display a window as shown in figure below:

Click "Start" button to start the service. Wait for few seconds or a minute, then open browser and type the given URL:  http://localhost:9200/  , you can the output as shown below if everything is fine.

The elastic service can also be run through windows services. Search for services on search box on windows 10 and click on it to open services window. Then search for "Elasticsearch" and right click on it and select "Start". Again, wait for few seconds and go to http://localhost:9200/ to check if service is running or not.

Read More

The fifth final heartbreak for CSK

Having played ten seasons of IPL, CSK holds impressive records of qualifying into the playoffs every time, reaching into final eight out of ten times. But at the same time it holds scary record of loosing most of the finals it reached. CSK now have lost 5 out of 8 IPL finals having won only 3 of them, most importantly three lose came against Mumbai Indians which CSK fans obviously finds it difficult to digest. MI has won four out of their five IPL finals, all coming against Dhoni (three against CSK and one against Rising Pune Supergaint).

In this IPL 2019, MI completely dominated CSK whether in league matches or qualifier. Infact, MI were only able to beat CSK at their home not only once but twice, one in league match and another in qualifier. So, MI reached final with three wins against CSK and at the same time CSK reached final with three loses against MI. But all CSK fans were hoping that CSK can't loose four match against MI in a single season, and CSK may win the final and guess what CSK almost won the final but that last ball wicket left CSK fans with nothing but heartbreak making it fifth final loss. This loss was difficult to digest than the last four finals which CSK lost because it was so near yet so far. CSK lost just by one run. The tragedy of 2017 repeated, the only difference being it was RPS at the receiving end in 2017 and this time it was CSK.

The 2019 IPL final was actually full of drama, mistakes being committed by both team. And as MSD said at post match presentation, it was like both teams were passing trophy to one another during match by committing mistakes and allowing the opposition to make come back in the game. But CSK made more mistakes than MI did and hence paid huge price for it by rewarding trophy to MI. There were many turning points for CSK. Many says the run out of MSD or the run out of Watson but I personally feel the turning point was the wicket of Faf du Plessis. Having known that the middle order has been completely exposed this season, he should have batted long. Though CSK were in a good position at the time fall of his wicket but he was hitting the ball very well. At the same time the upcoming batsmen had been struggling throughout the season except MSD, and MSD was obviously not going to win you matches every time. But none of us thought that MSD inning would end with a run out. It was infact first run out for MSD in IPL after 2012. So nothing was going in favour of CSK except three dropped catches of Watson to keep the hope of CSK alive till the last over in the night of final. But even Watson could not take CSK over the winning line and his fabulous inning came to an end with only two balls remaining and requiring four to win. Shardul Thakur managed only two runs loosing his wickets in the ball. He was playing his only third match this season so it was not easy for him as well. I feel Jadeja could have been sent to bat before Dwayne Bravo because he had been good with the bat this season and Bravo had rarely got any chance to bat and as a result Bravo scored 15 runs of 15 balls when the required rate was almost above 11. Also, I think Imran Tahir should have been brought little earlier to bowl and should have been allowed to complete his 4 overs quota as was leading wicket taker for CSK, infact ended as leading wicket taker of this season. But now, none of these arguments matter because the fact remains that CSK lost 2019 IPL final by 1 run.

At last as a CSK fan, the only good thing about the final was Thala saying "Hopefully I will return" next year for CSK because it is difficult to imagine CSK without him and IPL without CSK.

Read More

Easily migrate a Django DB from SQLite to MySQL.

Step1: 


Go to https://www.lfd.uci.edu/~gohlke/pythonlibs/#mysqlclient to download mysqlclient library and download the version which suits your python version. To check whether your installed python is 32/64 bits, simply search for python in search box in windows 10.

Warning: Though your pc is 64 bits, the python on your pc may be of 32 bits so download the mysqlclient file according to bits of python.
In my case, my python is 32 bits so I have installed the given file.














Step 2:

Now open django project and click on the terminal tab as shown in figure below (with red a mark):

Then copy the file download in step 1 to the folder of your django project (path marked by blue pen in figure above).

Step 3:


Run the given command:
pip install mysqlclient-1.4.2-cp37-cp37m-win32.whl

Here, the bold words are filename of the mysqlclient library so it may varies depending on which file did you download from the website. So use the file name of downloaded file in place of bold words in the command.

Finally, your install is successful.

Your output will be something like this:

(venv) G:\Programming\dj6>pip install mysqlclient-1.4.2-cp37-cp37m-win32.whl
Processing g:\programming\dj6\mysqlclient-1.4.2-cp37-cp37m-win32.whl
Installing collected packages: mysqlclient
Successfully installed mysqlclient-1.4.2

Here is the video for above process...

Step 4:

Open the setting.py file and change the section 

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
    }
}

into 

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'your database name',
        'USER': 'your username',
        'PASSWORD': 'your password',
        'HOST': 'localhost',
        'PORT': '3306',
    }
}

Step 5:

Use command in terminal of your django project:

python manage.py makemigrations

If it works then it is good but if it results in errors then follow the given below steps.

Step 6:

Alter the username and password of your MYSQL database server.

Open your MYSQL command line (search for mysql in search box and you can see this option). This write the given query:

ALTER USER 'root'@'localhost' IDENTIFIED BY 'your_new_password';

Warning: Don't forget the semicolon(;) at the end of query;

Step 7: 

Search for services in search box of windows 10 and open it. In newly  opened window search for MYSQL80 and click on it. Then click on Restart option on the left side of window.

Finally again use the command below in terminal of your django project:

python manage.py makemigrations

python manage.py migrate

Wait for sometime for migrating to complete.

Following above steps will do the job.

Read More

pip install mysqlclient error fixed. 100% working solution...

Step1: 

Go to https://www.lfd.uci.edu/~gohlke/pythonlibs/#mysqlclient to download mysqlclient library and download the version which suits your python version. To check whether your installed python is 32/64 bits, simply search for python in search box in windows 10.

Warning: Though your pc is 64 bits, the python on your pc may be of 32 bits so download the mysqlclient file according to bits of python.
In my case, my python is 32 bits so I have installed the given file.














Step 2:

Now open django project and click on the terminal tab as shown in figure below (with red a mark):

Then copy the file download in step 1 to the folder of your django project (path marked by blue pen in figure above).

Step 3:

Run the given command:
pip install mysqlclient-1.4.2-cp37-cp37m-win32.whl

Here, the bold words are filename of the mysqlclient library so it may varies depending on which file did you download from the website. So use the file name of downloaded file in place of bold words in the command.

Finally, your install is successful.

Your output will be something like this:

(venv) G:\Programming\dj6>pip install mysqlclient-1.4.2-cp37-cp37m-win32.whl
Processing g:\programming\dj6\mysqlclient-1.4.2-cp37-cp37m-win32.whl
Installing collected packages: mysqlclient
Successfully installed mysqlclient-1.4.2

The above helps to install mysqlclient in pycharm. If you want to install in python directly then simply download the file as in step 1 and open command prompt and navigate to the folder where the downloaded file is available. After that, run the command as shown in step 3. This will successfully finish your installation process.

Here is the video for above process...

Read More

Medical store management system with visualization of data (using visual C#.NET and MYSQL)

Send your query regarding this project through email.

Read More

How to make a simple picture gallery using lightbox library in web?

HTML code

<!DOCTYPE html>

<html>

<head>

<title>PicGallery</title>

<link rel="stylesheet" type="text/css" href="css/picgallery.css">

<link rel="stylesheet" type="text/css" href="css/lightbox.min.css">

<script type="text/javascript" src="js/lightbox-plus-jquery.min.js"></script>

</head>

<body>

<h1>Image Gallery</h1>

<div class="gallery">

<a href="gallery/1.jpg" data-lightbox="MyGallery" data-title="Red flower"><img src="gallery/1.jpg"></a>

<a href="gallery/2.jpg" data-lightbox="MyGallery" data-title="Car"><img src="gallery/2.jpg"></a>

<a href="gallery/3.jpg" data-lightbox="MyGallery"><img src="gallery/3.jpg"></a>

<a href="gallery/4.jpg" data-lightbox="MyGallery"><img src="gallery/4.jpg"></a>

<a href="gallery/5.jpg" data-lightbox="MyGallery"><img src="gallery/5.jpg"></a>

<a href="gallery/6.jpg" data-lightbox="MyGallery"><img src="gallery/6.jpg"></a>

<a href="gallery/7.jpg" data-lightbox="MyGallery"><img src="gallery/7.jpg"></a>

<a href="gallery/8.jpg" data-lightbox="MyGallery"><img src="gallery/8.jpg"></a>

</div>

</body>

</html>

CSS code

h1{
color: aqua;
text-align: center;
}

.gallery img{
width: 150px;
height: 150px;
padding: 20px;
transition: 1s;
}

.gallery img:hover{

transform: scale(1.5);
}


click here to download a complete file.

Read More

Design simple yet attractive log in form using HTML and CSS.

HTML code

<!DOCTYPE html>

<html>

<head>

<title>Feedback form</title>

<link rel="stylesheet" type="text/css" href="css/feedbackform.css">

</head>

<body>

<h3>Send your feedback...</h3>

<div class=blurform>

<div class="form1">

<form action="" method="get">

<br><br><label for="email">Your email id</label><br>

<input id="email" type="text" name="email"><br>

<label for="password">Your password</label><br>

<input id="password" type="text" name="password"><br>

<label for="comment">Comment</label><br>

<textarea id="comment" name="comment" rows="12" cols="40"></textarea><br>

<input type="checkbox" id="chkbx" name="chhkbx"><label for="chkbx">I completely agree to my statements.</label><br>

<input type="submit" value="Submit">

</form>

</div>

</div>

</body>

</html>

CSS code

input[type="text"]{

height: 25px;

width:250px;

font-size: large;

}

input[type="text"],textarea{

border:none;

margin-bottom: 20px;

margin-top: 8px;

background-color: #77aaff;

    color: white;

}

textarea{

width:350px;

}

input[type="submit"]{

cursor: pointer;

margin-top: 25px;

width: 120px;

height: 40px;

color: white;

background-color: #77aaff;

border: none;

margin-bottom: 30px;

}

label{

cursor: pointer;

 

}

.form1{

width:400px;

margin: auto;

background-color:white;

padding-left: 50px;

margin-top: 5px;

    border: 1px solid #77aaff;

    box-shadow: -7px 7px 8px #77aaff;

}

.blurform{

   width:400px;

margin: auto;

box-shadow: -7px 7px 300px #77aaff;

}

h3{

padding-left: 480px;

color:#77aaff;

}

Read More