Skip to content
Snippets Groups Projects
Commit ebef1a3e authored by Pieter van der Hijden's avatar Pieter van der Hijden
Browse files

Some filenames: spaces replaced with minus sign

parent 7707322c
No related branches found
No related tags found
No related merge requests found
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
# Generating a report on Aligning a Fab Lab with the UN SDGs # Generating a report on Aligning a Fab Lab with the UN SDGs
#### Author: Pieter van der Hijden (pvdh@sofos.nl) - "Fab Labs and SDGs" Working Group; Pieter van der Hijden (The Netherlands & Suriname) plus global team: Enrico Bassi (Italy), Vaneza Caycho Ñuflo (Peru), Nagwa ElNwishy (Egypt), Neville Govender (South Africa), Ted Hung (Taiwan), Arundhati Jadhav (India), Beno Juarez (Peru), Yogesh Kulkarni (India), Noksy Letsoalo (South Africa), Jean-Baptiste Natali (New Zealand), Wendy Neale (New Zealand), Mess Wright (USA) #### Author: Pieter van der Hijden (pvdh@sofos.nl) - "Fab Labs and SDGs" Working Group; Pieter van der Hijden (The Netherlands & Suriname) plus global team: Enrico Bassi (Italy), Vaneza Caycho Ñuflo (Peru), Nagwa ElNwishy (Egypt), Neville Govender (South Africa), Ted Hung (Taiwan), Arundhati Jadhav (India), Beno Juarez (Peru), Yogesh Kulkarni (India), Noksy Letsoalo (South Africa), Jean-Baptiste Natali (New Zealand), Wendy Neale (New Zealand), Mess Wright (USA)
#### This work is licensed under a Creative Commons Attribution 4.0 International License; https://creativecommons.org/licenses/by/4.0/. #### This work is licensed under a Creative Commons Attribution 4.0 International License; https://creativecommons.org/licenses/by/4.0/.
### Purpose ### Purpose
As a service to the individual fablabs we want to generate a report on a specific fab lab and its alignment with the UN Sustrainable Development Goals (SDGs). The information included is either generic in nature, or derived from various databases (fablabs basic data from fablabs.io, fablabs sdg profiles from sdg profile registration spreadsheet). As a service to the individual fablabs we want to generate a report on a specific fab lab and its alignment with the UN Sustrainable Development Goals (SDGs). The information included is either generic in nature, or derived from various databases (fablabs basic data from fablabs.io, fablabs sdg profiles from sdg profile registration spreadsheet).
### Table of Contents ### Table of Contents
1. Install packages 1. Install packages
1. Import libraries 1. Import libraries
1. Prepare dataframes 1. Prepare dataframes
* Prepare dataframe profiles from gitlab.fabcloud.org * Prepare dataframe profiles from gitlab.fabcloud.org
* Prepare dataframe fablabs from fablabs.io * Prepare dataframe fablabs from fablabs.io
1. Merge profiles with fablabs (left) 1. Merge profiles with fablabs (left)
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## Install packages ## Install packages (optional)
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
# Install packages # Install packages
!sudo pip3 install pandas !sudo pip3 install pandas
!sudo pip3 install openpyxl !sudo pip3 install openpyxl
``` ```
%% Output %% Output
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (1.2.3) Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (1.2.3)
Requirement already satisfied: numpy>=1.16.5 in /usr/local/lib/python3.7/dist-packages (from pandas) (1.20.1) Requirement already satisfied: numpy>=1.16.5 in /usr/local/lib/python3.7/dist-packages (from pandas) (1.20.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/lib/python3/dist-packages (from pandas) (2.7.3) Requirement already satisfied: python-dateutil>=2.7.3 in /usr/lib/python3/dist-packages (from pandas) (2.7.3)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas) (2021.1) Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas) (2021.1)
WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Requirement already satisfied: openpyxl in /usr/local/lib/python3.7/dist-packages (3.0.7) Requirement already satisfied: openpyxl in /usr/local/lib/python3.7/dist-packages (3.0.7)
Requirement already satisfied: et-xmlfile in /usr/local/lib/python3.7/dist-packages (from openpyxl) (1.0.1) Requirement already satisfied: et-xmlfile in /usr/local/lib/python3.7/dist-packages (from openpyxl) (1.0.1)
WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## Install newest pip (optional) ## Install newest pip (optional)
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
!sudo /usr/bin/python3 -m pip install --upgrade pip !sudo /usr/bin/python3 -m pip install --upgrade pip
``` ```
%% Output %% Output
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Requirement already satisfied: pip in /usr/local/lib/python3.7/dist-packages (21.1.1) Requirement already satisfied: pip in /usr/local/lib/python3.7/dist-packages (21.1.1)
WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## Install libraries ## Install libraries
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
import pandas as pd import pandas as pd
import datetime import datetime
import os import os
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## Prepare dataframe 'profiles' from https://gitlab.fabcloud.org ## Prepare dataframe 'profiles' from https://gitlab.fabcloud.org
Options Options
* Read xlsx data from gitlab.fabcloud.org to pandas dataframe 'profiles' * Read xlsx data from gitlab.fabcloud.org to pandas dataframe 'profiles'
* Save dataframe 'profiles' to data/profiles.csv * Save dataframe 'profiles' to data/profiles.csv
* Read dataframe 'profiles' from data/profiles.csv * Read dataframe 'profiles' from data/profiles.csv
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
# Get data from gitlab.fabcloud.org # Get data from gitlab.fabcloud.org
# Read XLSX sheet with fab lab sdg profiles # Read XLSX sheet with fab lab sdg profiles
url = 'https://gitlab.fabcloud.org/fl-management/fl-sdgs/sdg-workshop/-/raw/master/' url = 'https://gitlab.fabcloud.org/fl-management/fl-sdgs/sdg-workshop/-/raw/master/'
url = url + 'aggregated-results/FabLab-SDGProfile-Registration.xlsx' url = url + 'aggregated-results/FabLab-SDGProfile-Registration.xlsx'
timestamp = datetime.datetime.utcnow() timestamp = datetime.datetime.utcnow()
profiles=pd.read_excel(url, 'processing', index_col=27) #profiles=pd.read_excel(url, 'processing', index_col=27)
# Write dataframe to local storage # Write dataframe to local storage
if not os.path.exists('data'): if not os.path.exists('data'):
os.makedirs('data') os.makedirs('data')
# Write dataframe to csv-file # Write dataframe to csv-file
profiles.to_csv('data/profiles.csv') #profiles.to_csv('data/profiles.csv')
# Read dataframe from local storage # Read dataframe from local storage
profiles = pd.read_csv('data/profiles.csv') profiles = pd.read_csv('data/profiles.csv')
# log number of objects and number of fields # log number of objects and number of fields
if not os.path.exists('output'): if not os.path.exists('output'):
os.makedirs('output') os.makedirs('output')
line = str(timestamp) + ' - ' + url + ' - ' + str(profiles.shape) line = str(timestamp) + ' - ' + url + ' - ' + str(profiles.shape)
f = open('output/log.txt', 'a') f = open('output/log.txt', 'a')
f.write(line) f.write(line)
#f.close() #f.close()
profiles.shape profiles.shape
``` ```
%% Output %% Output
(244, 29) (244, 29)
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## Clean dataframe 'profiles' ## Clean dataframe 'profiles'
* Delete verbose and superfluous columns * Delete verbose and superfluous columns
* Skip empty rows * Skip empty rows
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
# Clean dataframe 'profiles' # Clean dataframe 'profiles'
# Delete verbose and superfluous columns 6-7 # Delete verbose and superfluous columns 6-7
del profiles['Enter your SDG profile by ticking 2-4 SDGs'] del profiles['Enter your SDG profile by ticking 2-4 SDGs']
del profiles['One more thing!'] del profiles['One more thing!']
# Change column names
profiles = profiles.rename(columns={'The name of your fab lab':'fab-name'})
profiles = profiles.rename(columns={'Country of your fab lab':'fab-country'})
profiles = profiles.rename(columns={'URL of your fab lab page on www.fablabs.io':'fablabs.io-url'})
for s in range(17):
profiles = profiles.rename(columns={'SDG ' + str(s+1):'SDG-' + f'{(s+1):02d}'}) # SDG columns 2 digits 0 spaces
profiles.columns = profiles.columns.str.replace(' ', '') # remove spaces in column names
profiles.columns = profiles.columns.str.lower() # convert column names to lower case
profiles.columns = profiles.columns.str.replace('sdg', 'SDG') # remove spaces in column names
profiles.shape profiles.shape
``` ```
%% Output %% Output
(244, 27) (244, 27)
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
profiles.info()
```
%% Output
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 244 entries, 0 to 243
Data columns (total 27 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 slug 231 non-null object
1 tijdstempel 238 non-null object
2 e-mailadres 238 non-null object
3 yourfullname 238 non-null object
4 fab-name 238 non-null object
5 fab-country 237 non-null object
6 fablabs.io-url 238 non-null object
7 SDG-01 9 non-null object
8 SDG-02 13 non-null object
9 SDG-03 26 non-null object
10 SDG-04 190 non-null object
11 SDG-05 75 non-null object
12 SDG-06 14 non-null object
13 SDG-07 33 non-null object
14 SDG-08 68 non-null object
15 SDG-09 156 non-null object
16 SDG-10 24 non-null object
17 SDG-11 82 non-null object
18 SDG-12 102 non-null object
19 SDG-13 24 non-null object
20 SDG-14 7 non-null object
21 SDG-15 18 non-null object
22 SDG-16 2 non-null object
23 SDG-17 42 non-null object
24 summary 244 non-null object
25 status 244 non-null object
26 check 221 non-null object
dtypes: object(27)
memory usage: 25.8+ KB
%% Cell type:code id: tags:
``` python
# Clean dataframe 'profiles' # Clean dataframe 'profiles'
# Skip empty rows # Skip empty rows
profiles = profiles.loc[profiles['Tijdstempel'].notna()] profiles = profiles.loc[profiles['tijdstempel'].notna()]
profiles.shape profiles.shape
``` ```
%% Output %% Output
(238, 27) (238, 27)
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## Prepare dataframe 'fablabs' from https://fablabs.io ## Prepare dataframe 'fablabs' from https://fablabs.io
Options Options
* Read json data from fablabs.io to pandas dataframe 'fablabs' * Read json data from fablabs.io to pandas dataframe 'fablabs'
* Save dataframe 'fablabs' to data/fablabs.csv * Save dataframe 'fablabs' to data/fablabs.csv
* Read dataframe 'fablabs' from data/fablabs.csv * Read dataframe 'fablabs' from data/fablabs.csv
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
# Get data from fablabs.io web site # Get data from fablabs.io web site
# Read JSON from website and convert to Dataframe # Read JSON from website and convert to Dataframe
url = "https://fablabs.io/labs.json?class=btn+btn-primary" url = "https://fablabs.io/labs.json?class=btn+btn-primary"
timestamp = datetime.datetime.utcnow() timestamp = datetime.datetime.utcnow()
fablabs = pd.read_json(url) #fablabs = pd.read_json(url)
fablabs = fablabs.set_index('slug') #fablabs = fablabs.set_index('slug')
# Write dataframe to local storage # Write dataframe to local storage
if not os.path.exists('data'): if not os.path.exists('data'):
os.makedirs('data') os.makedirs('data')
# Write dataframe to csv-file # Write dataframe to csv-file
fablabs.to_csv('data/fablabs.csv') #fablabs.to_csv('data/fablabs.csv')
# Read dataframe from local storage # Read dataframe from local storage
#fablabs = pd.read_csv('data/fablabs.csv') fablabs = pd.read_csv('data/fablabs.csv')
# log number of objects and number of fields # log number of objects and number of fields
if not os.path.exists('output'): if not os.path.exists('output'):
os.makedirs('output') os.makedirs('output')
line = str(timestamp) + ' - ' + url + ' - ' + str(fablabs.shape) line = str(timestamp) + ' - ' + url + ' - ' + str(fablabs.shape)
f = open('output/log.txt', 'a') f = open('output/log.txt', 'a')
f.write(line) f.write(line)
#f.close() #f.close()
fablabs.shape fablabs.shape
``` ```
%% Output %% Output
(2015, 22) (2015, 23)
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## Merge profiles data with fablabs data ## Merge profiles data with fablabs data
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
profiles = pd.merge(profiles, fablabs, how="left", on="slug") profiles = pd.merge(profiles, fablabs, how="left", on="slug")
profiles.shape profiles.shape
``` ```
%% Output %% Output
(238, 49) (238, 49)
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## Create table: SDG preferences
%% Cell type:code id: tags:
``` python
profiles.columns
```
%% Output
Index(['slug', 'tijdstempel', 'e-mailadres', 'yourfullname', 'fab-name',
'fab-country', 'fablabs.io-url', 'SDG-01', 'SDG-02', 'SDG-03', 'SDG-04',
'SDG-05', 'SDG-06', 'SDG-07', 'SDG-08', 'SDG-09', 'SDG-10', 'SDG-11',
'SDG-12', 'SDG-13', 'SDG-14', 'SDG-15', 'SDG-16', 'SDG-17', 'summary',
'status', 'check', 'id', 'name', 'kind_name', 'parent_id', 'blurb',
'description', 'avatar_url', 'header_url', 'address_1', 'address_2',
'city', 'county', 'postal_code', 'country_code', 'latitude',
'longitude', 'address_notes', 'phone', 'email', 'capabilities',
'activity_status', 'links'],
dtype='object')
%% Cell type:code id: tags:
``` python
# select rows with status = 'green' and columns sdg1-17
# create list of sdg column names
sdgs = []
for s in range(17):
sdgs = sdgs + ["SDG-" + f'{(s+1):02d}']
selprofiles = profiles[profiles['status'] == 'green'][sdgs] # select rows flagged 'green' and columns SDG1-17
selseries = selprofiles.count() # create series with sdg counts 1-17
mytable = selseries.to_frame() # convert series to frame
mytable.head(20)
```
%% Output
0
SDG-01 8
SDG-02 12
SDG-03 23
SDG-04 172
SDG-05 70
SDG-06 12
SDG-07 30
SDG-08 63
SDG-09 148
SDG-10 22
SDG-11 79
SDG-12 96
SDG-13 24
SDG-14 5
SDG-15 16
SDG-16 2
SDG-17 35
%% Cell type:code id: tags:
``` python
seltable = selseries.to_frame()
seltable.info()
```
%% Output
<class 'pandas.core.frame.DataFrame'>
Index: 17 entries, SDG-01 to SDG-17
Data columns (total 1 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 0 17 non-null int64
dtypes: int64(1)
memory usage: 204.0+ bytes
%% Cell type:markdown id: tags:
## State-of-affairs ## State-of-affairs
* input profiles, input fablabs and merging profiles+fablabs ok * current: create table with sdg preferences: SDG-code, SDG-label, Count, Percentage
* next: create piechart based on table
* next: export table and piechart as images
......
No preview for this file type
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment