Complete Pandas Tutorial in Python
අවුල් සහගත CSV file එකක්, model එකකට ඉගෙන ගන්න පුළුවන් දෙයක් බවට හරවන tool එක තමයි Pandas. සැබෑ ML projects ගොඩක් පටන් ගන්නෙ model එකෙන් නෙවෙයි, මෙතනින්.
Pandas is the tool that turns a messy CSV file into something a model can actually learn from. Almost every real ML project starts here, not with the model.
Lesson 10 of 64 · Part 1 · Python & Data Basics · Python Libraries · bilingual notes (English + සිංහල)
Why this matters. Pandas is the tool that turns a messy CSV file into something a model can actually learn from. Almost every real ML project starts here, not with the model.
සිංහලෙන්: අවුල් සහගත CSV file එකක්, model එකකට ඉගෙන ගන්න පුළුවන් දෙයක් බවට හරවන tool එක තමයි Pandas. සැබෑ ML projects ගොඩක් පටන් ගන්නෙ model එකෙන් නෙවෙයි, මෙතනින්.
What you'll use this for. You will use read_csv, .head(), .describe(), and .dropna() in literally every single project notebook from here on.
සිංහලෙන්: read_csv, .head(), .describe(), .dropna() — මේ ඔක්කොම මින් ඉස්සරහට එන හැම project notebook එකකම පාවිච්චි වෙනවා.
Key principles
- A DataFrame is a spreadsheet Python can compute on. Rows are records (one patient, one house), columns are features (age, price) — the exact shape scikit-learn expects.
DataFrame එකක් කියන්නෙ Python එකට calculate කරන්න පුළුවන් spreadsheet එකක් - Real data is never clean — that's the whole point of this lesson. Missing values, wrong types, duplicate rows — Pandas gives you
.isnull(),.fillna(),.drop_duplicates()to fix all of it before it reaches your model.
සැබෑ data කවදාවත් පිරිසිදු නෑ — මේ පාඩමේ මුළු අදහසම ඒකයි
Walkthrough
Pandas Library:
Useful for Data Processing & Analysis
Pandas Data Frame:
Pandas DataFrame is two-dimensional tabular data structure with labeled axes (rows and columns).
ExamplePython
# importing the pandas library
import pandas as pd
import numpy as npCreaating a Pandas DataFrame
ExamplePython
# importing the boston house price data
from sklearn.datasets import load_bostonExamplePython
boston_dataset = load_boston()ExamplePython
type(boston_dataset)Output
Output
sklearn.utils.BunchExamplePython
print(boston_dataset)Output
Output
{'data': array([[6.3200e-03, 1.8000e+01, 2.3100e+00, ..., 1.5300e+01, 3.9690e+02,
4.9800e+00],
[2.7310e-02, 0.0000e+00, 7.0700e+00, ..., 1.7800e+01, 3.9690e+02,
9.1400e+00],
[2.7290e-02, 0.0000e+00, 7.0700e+00, ..., 1.7800e+01, 3.9283e+02,
4.0300e+00],
...,
[6.0760e-02, 0.0000e+00, 1.1930e+01, ..., 2.1000e+01, 3.9690e+02,
5.6400e+00],
[1.0959e-01, 0.0000e+00, 1.1930e+01, ..., 2.1000e+01, 3.9345e+02,
6.4800e+00],
[4.7410e-02, 0.0000e+00, 1.1930e+01, ..., 2.1000e+01, 3.9690e+02,
7.8800e+00]]), 'target': array([24. , 21.6, 34.7, 33.4, 36.2, 28.7, 22.9, 27.1, 16.5, 18.9, 15. ,
18.9, 21.7, 20.4, 18.2, 19.9, 23.1, 17.5, 20.2, 18.2, 13.6, 19.6,
15.2, 14.5, 15.6, 13.9, 16.6, 14.8, 18.4, 21. , 12.7, 14.5, 13.2,
13.1, 13.5, 18.9, 20. , 21. , 24.7, 30.8, 34.9, 26.6, 25.3, 24.7,
21.2, 19.3, 20. , 16.6, 14.4, 19.4, 19.7, 20.5, 25. , 23.4, 18.9,
35.4, 24.7, 31.6, 23.3, 19.6, 18.7, 16. , 22.2, 25. , 33. , 23.5,
… (41 more lines)ExamplePython
# pandas DataFrame
boston_df = pd.DataFrame(boston_dataset.data, columns = boston_dataset.feature_names)ExamplePython
boston_df.head()| CRIM | ZN | INDUS | CHAS | NOX | RM | AGE | … | |
|---|---|---|---|---|---|---|---|---|
| 0 | 0.00632 | 18.0 | 2.31 | 0.0 | 0.538 | 6.575 | 65.2 | … |
| 1 | 0.02731 | 0.0 | 7.07 | 0.0 | 0.469 | 6.421 | 78.9 | … |
| 2 | 0.02729 | 0.0 | 7.07 | 0.0 | 0.469 | 7.185 | 61.1 | … |
| 3 | 0.03237 | 0.0 | 2.18 | 0.0 | 0.458 | 6.998 | 45.8 | … |
| 4 | 0.06905 | 0.0 | 2.18 | 0.0 | 0.458 | 7.147 | 54.2 | … |
ExamplePython
boston_df.shapeOutput
Output
(506, 13)ExamplePython
type(boston_df)Output
Output
pandas.core.frame.DataFrameImporting the data from a CSV file to a pandas DataFrame
ExamplePython
# csv file to pandas df
diabetes_df = pd.read_csv('/content/diabetes.csv')ExamplePython
type(diabetes_df)Output
Output
pandas.core.frame.DataFrameExamplePython
diabetes_df.head()| Pregnancies | Glucose | BloodPressure | SkinThickness | Insulin | BMI | DiabetesPedigreeFunction | … | |
|---|---|---|---|---|---|---|---|---|
| 0 | 6 | 148 | 72 | 35 | 0 | 33.6 | 0.627 | … |
| 1 | 1 | 85 | 66 | 29 | 0 | 26.6 | 0.351 | … |
| 2 | 8 | 183 | 64 | 0 | 0 | 23.3 | 0.672 | … |
| 3 | 1 | 89 | 66 | 23 | 94 | 28.1 | 0.167 | … |
| 4 | 0 | 137 | 40 | 35 | 168 | 43.1 | 2.288 | … |
ExamplePython
diabetes_df.shapeOutput
Output
(768, 9)Loading the data from a excel file to a Pandas DataFrame:
pd.read_excel('file path')
Exporting a DataFrame to a csv file
ExamplePython
boston_df.to_csv('boston.csv')Exporting the Pandas DataFrame to an excel File:
df.to_excel('filename')
ExamplePython
# creating a DatFrame with random values
random_df = pd.DataFrame(np.random.rand(20,10))ExamplePython
random_df.head()| 0 | 1 | 2 | 3 | 4 | 5 | 6 | … | |
|---|---|---|---|---|---|---|---|---|
| 0 | 0.978586 | 0.798457 | 0.756065 | 0.574559 | 0.731437 | 0.976397 | 0.865563 | … |
| 1 | 0.078777 | 0.295596 | 0.608408 | 0.310086 | 0.326199 | 0.086600 | 0.549180 | … |
| 2 | 0.172873 | 0.971198 | 0.592191 | 0.482904 | 0.171601 | 0.981757 | 0.268020 | … |
| 3 | 0.812212 | 0.612494 | 0.011446 | 0.817039 | 0.076837 | 0.712891 | 0.929890 | … |
| 4 | 0.629319 | 0.046347 | 0.918180 | 0.317385 | 0.692454 | 0.793890 | 0.016952 | … |
ExamplePython
random_df.shapeOutput
Output
(20, 10)Inspecting a DataFrame
ExamplePython
#finding the number of rows & columns
boston_df.shapeOutput
Output
(506, 13)ExamplePython
# first 5 rows in a DataFrame
boston_df.head()| CRIM | ZN | INDUS | CHAS | NOX | RM | AGE | … | |
|---|---|---|---|---|---|---|---|---|
| 0 | 0.00632 | 18.0 | 2.31 | 0.0 | 0.538 | 6.575 | 65.2 | … |
| 1 | 0.02731 | 0.0 | 7.07 | 0.0 | 0.469 | 6.421 | 78.9 | … |
| 2 | 0.02729 | 0.0 | 7.07 | 0.0 | 0.469 | 7.185 | 61.1 | … |
| 3 | 0.03237 | 0.0 | 2.18 | 0.0 | 0.458 | 6.998 | 45.8 | … |
| 4 | 0.06905 | 0.0 | 2.18 | 0.0 | 0.458 | 7.147 | 54.2 | … |
ExamplePython
# last 5 rows of the DataFrame
boston_df.tail()| CRIM | ZN | INDUS | CHAS | NOX | RM | AGE | … | |
|---|---|---|---|---|---|---|---|---|
| 501 | 0.06263 | 0.0 | 11.93 | 0.0 | 0.573 | 6.593 | 69.1 | … |
| 502 | 0.04527 | 0.0 | 11.93 | 0.0 | 0.573 | 6.120 | 76.7 | … |
| 503 | 0.06076 | 0.0 | 11.93 | 0.0 | 0.573 | 6.976 | 91.0 | … |
| 504 | 0.10959 | 0.0 | 11.93 | 0.0 | 0.573 | 6.794 | 89.3 | … |
| 505 | 0.04741 | 0.0 | 11.93 | 0.0 | 0.573 | 6.030 | 80.8 | … |
ExamplePython
# informations about the DataFrame
boston_df.info()Output
Output
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 506 entries, 0 to 505
Data columns (total 13 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 CRIM 506 non-null float64
1 ZN 506 non-null float64
2 INDUS 506 non-null float64
3 CHAS 506 non-null float64
4 NOX 506 non-null float64
5 RM 506 non-null float64
6 AGE 506 non-null float64
7 DIS 506 non-null float64
8 RAD 506 non-null float64
9 TAX 506 non-null float64
10 PTRATIO 506 non-null float64
11 B 506 non-null float64
12 LSTAT 506 non-null float64
… (2 more lines)ExamplePython
# finding the number of missing values
boston_df.isnull().sum()Output
Output
CRIM 0
ZN 0
INDUS 0
CHAS 0
NOX 0
RM 0
AGE 0
DIS 0
RAD 0
TAX 0
PTRATIO 0
B 0
LSTAT 0
dtype: int64ExamplePython
# diabetes dataframe
diabetes_df.head()| Pregnancies | Glucose | BloodPressure | SkinThickness | Insulin | BMI | DiabetesPedigreeFunction | … | |
|---|---|---|---|---|---|---|---|---|
| 0 | 6 | 148 | 72 | 35 | 0 | 33.6 | 0.627 | … |
| 1 | 1 | 85 | 66 | 29 | 0 | 26.6 | 0.351 | … |
| 2 | 8 | 183 | 64 | 0 | 0 | 23.3 | 0.672 | … |
| 3 | 1 | 89 | 66 | 23 | 94 | 28.1 | 0.167 | … |
| 4 | 0 | 137 | 40 | 35 | 168 | 43.1 | 2.288 | … |
ExamplePython
# counting the values based on the labels
diabetes_df.value_counts('Outcome')Output
Output
Outcome
0 500
1 268
dtype: int64ExamplePython
# group the values based on the mean
diabetes_df.groupby('Outcome').mean()| Pregnancies | Glucose | BloodPressure | SkinThickness | Insulin | BMI | DiabetesPedigreeFunction | Age | |
|---|---|---|---|---|---|---|---|---|
| Outcome | ||||||||
| 0 | 3.298000 | 109.980000 | 68.184000 | 19.664000 | 68.792000 | 30.304200 | 0.429734 | 31.190000 |
| 1 | 4.865672 | 141.257463 | 70.824627 | 22.164179 | 100.335821 | 35.142537 | 0.550500 | 37.067164 |
Statistical Measures
ExamplePython
# count or number of values
boston_df.count()Output
Output
CRIM 506
ZN 506
INDUS 506
CHAS 506
NOX 506
RM 506
AGE 506
DIS 506
RAD 506
TAX 506
PTRATIO 506
B 506
LSTAT 506
dtype: int64ExamplePython
# mean value - column wise
boston_df.mean()Output
Output
CRIM 3.613524
ZN 11.363636
INDUS 11.136779
CHAS 0.069170
NOX 0.554695
RM 6.284634
AGE 68.574901
DIS 3.795043
RAD 9.549407
TAX 408.237154
PTRATIO 18.455534
B 356.674032
LSTAT 12.653063
dtype: float64ExamplePython
# standard deviation - column wise
boston_df.std()Output
Output
CRIM 8.601545
ZN 23.322453
INDUS 6.860353
CHAS 0.253994
NOX 0.115878
RM 0.702617
AGE 28.148861
DIS 2.105710
RAD 8.707259
TAX 168.537116
PTRATIO 2.164946
B 91.294864
LSTAT 7.141062
dtype: float64ExamplePython
# minimum value
boston_df.min()Output
Output
CRIM 0.00632
ZN 0.00000
INDUS 0.46000
CHAS 0.00000
NOX 0.38500
RM 3.56100
AGE 2.90000
DIS 1.12960
RAD 1.00000
TAX 187.00000
PTRATIO 12.60000
B 0.32000
LSTAT 1.73000
dtype: float64ExamplePython
# maximum value
boston_df.max()Output
Output
CRIM 88.9762
ZN 100.0000
INDUS 27.7400
CHAS 1.0000
NOX 0.8710
RM 8.7800
AGE 100.0000
DIS 12.1265
RAD 24.0000
TAX 711.0000
PTRATIO 22.0000
B 396.9000
LSTAT 37.9700
dtype: float64ExamplePython
# all the statistical measures about the dataframe
boston_df.describe()| CRIM | ZN | INDUS | CHAS | NOX | RM | AGE | … | |
|---|---|---|---|---|---|---|---|---|
| count | 506.000000 | 506.000000 | 506.000000 | 506.000000 | 506.000000 | 506.000000 | 506.000000 | … |
| mean | 3.613524 | 11.363636 | 11.136779 | 0.069170 | 0.554695 | 6.284634 | 68.574901 | … |
| std | 8.601545 | 23.322453 | 6.860353 | 0.253994 | 0.115878 | 0.702617 | 28.148861 | … |
| min | 0.006320 | 0.000000 | 0.460000 | 0.000000 | 0.385000 | 3.561000 | 2.900000 | … |
| 25% | 0.082045 | 0.000000 | 5.190000 | 0.000000 | 0.449000 | 5.885500 | 45.025000 | … |
| 50% | 0.256510 | 0.000000 | 9.690000 | 0.000000 | 0.538000 | 6.208500 | 77.500000 | … |
… 2 more rows
Manipulating a DataFrame
ExamplePython
# adding a column to a dataframe
boston_df['Price'] = boston_dataset.targetExamplePython
boston_df.head()| CRIM | ZN | INDUS | CHAS | NOX | RM | AGE | … | |
|---|---|---|---|---|---|---|---|---|
| 0 | 0.00632 | 18.0 | 2.31 | 0.0 | 0.538 | 6.575 | 65.2 | … |
| 1 | 0.02731 | 0.0 | 7.07 | 0.0 | 0.469 | 6.421 | 78.9 | … |
| 2 | 0.02729 | 0.0 | 7.07 | 0.0 | 0.469 | 7.185 | 61.1 | … |
| 3 | 0.03237 | 0.0 | 2.18 | 0.0 | 0.458 | 6.998 | 45.8 | … |
| 4 | 0.06905 | 0.0 | 2.18 | 0.0 | 0.458 | 7.147 | 54.2 | … |
ExamplePython
# removing a row
boston_df.drop(index=0, axis=0)| CRIM | ZN | INDUS | CHAS | NOX | RM | AGE | … | |
|---|---|---|---|---|---|---|---|---|
| 1 | 0.02731 | 0.0 | 7.07 | 0.0 | 0.469 | 6.421 | 78.9 | … |
| 2 | 0.02729 | 0.0 | 7.07 | 0.0 | 0.469 | 7.185 | 61.1 | … |
| 3 | 0.03237 | 0.0 | 2.18 | 0.0 | 0.458 | 6.998 | 45.8 | … |
| 4 | 0.06905 | 0.0 | 2.18 | 0.0 | 0.458 | 7.147 | 54.2 | … |
| 5 | 0.02985 | 0.0 | 2.18 | 0.0 | 0.458 | 6.430 | 58.7 | … |
| ... | ... | ... | ... | ... | ... | ... | ... | … |
… 5 more rows
ExamplePython
# drop a column
boston_df.drop(columns='ZN', axis=1)| CRIM | INDUS | CHAS | NOX | RM | AGE | DIS | … | |
|---|---|---|---|---|---|---|---|---|
| 0 | 0.00632 | 2.31 | 0.0 | 0.538 | 6.575 | 65.2 | 4.0900 | … |
| 1 | 0.02731 | 7.07 | 0.0 | 0.469 | 6.421 | 78.9 | 4.9671 | … |
| 2 | 0.02729 | 7.07 | 0.0 | 0.469 | 7.185 | 61.1 | 4.9671 | … |
| 3 | 0.03237 | 2.18 | 0.0 | 0.458 | 6.998 | 45.8 | 6.0622 | … |
| 4 | 0.06905 | 2.18 | 0.0 | 0.458 | 7.147 | 54.2 | 6.0622 | … |
| ... | ... | ... | ... | ... | ... | ... | ... | … |
… 5 more rows
ExamplePython
# locating a row using the index value
boston_df.iloc[2]Output
Output
CRIM 0.02729
ZN 0.00000
INDUS 7.07000
CHAS 0.00000
NOX 0.46900
RM 7.18500
AGE 61.10000
DIS 4.96710
RAD 2.00000
TAX 242.00000
PTRATIO 17.80000
B 392.83000
LSTAT 4.03000
Price 34.70000
Name: 2, dtype: float64ExamplePython
# locating a particular column
print(boston_df.iloc[:,0]) # first column
print(boston_df.iloc[:,1]) # second column
print(boston_df.iloc[:,2]) # third column
print(boston_df.iloc[:,-1]) # last columnOutput
Output
0 0.00632
1 0.02731
2 0.02729
3 0.03237
4 0.06905
...
501 0.06263
502 0.04527
503 0.06076
504 0.10959
505 0.04741
Name: CRIM, Length: 506, dtype: float64
0 18.0
1 0.0
2 0.0
3 0.0
4 0.0
...
… (30 more lines)Correlation:
- Positive Correlation
- Negative Correlation
ExamplePython
boston_df.corr()| CRIM | ZN | INDUS | CHAS | NOX | RM | AGE | … | |
|---|---|---|---|---|---|---|---|---|
| CRIM | 1.000000 | -0.200469 | 0.406583 | -0.055892 | 0.420972 | -0.219247 | 0.352734 | … |
| ZN | -0.200469 | 1.000000 | -0.533828 | -0.042697 | -0.516604 | 0.311991 | -0.569537 | … |
| INDUS | 0.406583 | -0.533828 | 1.000000 | 0.062938 | 0.763651 | -0.391676 | 0.644779 | … |
| CHAS | -0.055892 | -0.042697 | 0.062938 | 1.000000 | 0.091203 | 0.091251 | 0.086518 | … |
| NOX | 0.420972 | -0.516604 | 0.763651 | 0.091203 | 1.000000 | -0.302188 | 0.731470 | … |
| RM | -0.219247 | 0.311991 | -0.391676 | 0.091251 | -0.302188 | 1.000000 | -0.240265 | … |
… 8 more rows
Keep going: ← Lesson 9 · Lesson 11 → · Track it in the Self Guide
Finished this one?
Part of Self Guide · Stage 13: ML 1 — Python & Data Basics. Progress is saved in this browser.
