DPS909 & OSD600 Fall 2019

Week 1

• Course introduction
  • DPS909
  • OSD600
  • How to be successful in this course

• Labs
  • Weekly labs, typically done in class
  • Labs are due on the Friday of the week they are assigned by midnight
  • Marked using Pass/Fail scheme
  • All labs must be completed to pass the course
  • Lab 1 is available now

• Releases
  • 4 releases, some with multiple bugs/PRs required, including participating in Hacktoberfest 2019
  • Due Dates: Sept 20, Oct 31, Nov 20, Dec 6
  • Chance to work on real code, real projects
  • Big learning curve, lots of time required
  • Amazing chance to gain experience, network, build your skills and resume
  • Work with new and emerging technologies, gain exposure to tech outside the classroom

• Discussion/Readings
  • Copyright (Copyright in Canada video)
    • IANAL
    • Who created it, "owns" it.
    • Set of exclusive rights granted to the work's creator
    • "The right to copy," to produce or reproduce a work or substantial portion thereof
    • Copyright is automatic when a work is created, you don't have to register it.
    • Copyright in Canada
    • Copyright Guide
    • In a software project, there can be many copyright holders (e.g., many contributors), or all contributors may assign their copyright to the project (e.g., CLA, which we'll cover later)

• What is Open Source?
  • The Open Source Definition
  • Microsoft buys GitHub for 7.5 Billion - "We're all in on open source"
    • https://twitter.com/asynchio/status/1082702841964244992
  • Julia Evans, "How to teach yourself hard things"

Week 2

• Blogging
  • Add bio/profile info as you feel comfortable, including links to GitHub, social media, etc.
  • Blog Post Tips:
    • Use blog post titles that help a reader (or Google searcher) to know whether this is useful info to them
    • Include links: a blog should connect different resources and ideas through your experience and learning
    • Write in sections. Try to avoid a wall of text, with only a single paragraph. Consider using sub-headings, shorter paragraphs
    • Use formatting for source code.
  • One good source of blog posts on open source and software development is Hacker News. Some recent examples to look at for style:
    • https://css-tricks.com/how-to-contribute-to-an-open-source-project/
    • https://antoinevastel.com/javascript/2019/09/09/improving-obfuscator.html
    • https://localghost.dev/2019/09/everything-i-googled-in-a-week-as-a-professional-software-engineer/
    • https://randomascii.wordpress.com/2019/09/08/taskbar-latency-and-kernel-calls/

• Licenses
  • Rights, privileges, responsibilities, etc. applicable to someone other than the work's creator
  • "Terms and Conditions"
  • These must be granted by a copyright holder

• No License
  • What can you do with code you find that has no license?
  • what can I, can't I do?

• Proprietary Licenses
  • Spotify End User Agreement
  • Microsoft Windows
  • iOS 12 (pdf)

• Public Domain
  • SQLite, which is now used by literally everybody, see http://www.sqlite.org/famous.html
  • Unlicense

• BSD License
  • Family of Licenses, including 2-Clause BSD, 3-Clause BSD (aka New BDS), 4-Clause BSD
  • "Why you should use a BSD style license for your Open Source Project"
  • BSD Licenses code is usually compatible with other open/closed code, when you want to mix them.
  • Example software projects licensed under the BSD License:
    • Many Google Projects, including V8, Chromium, Go
    • D3.js
    • Django
    • nginx
    • Vi
  • Summary:
    • You need to retain the license and copyright notice
    • You can use it commercially or non-commercially (privately)
    • You can distribute it freely
    • You can modify it freely

• Discussion of projects found for Lab 1
• Lab 2
• Release 0.1 due Mon, Sept 23

Week 3

• Readings/Resources
  • Courses on Lynda
    • Git Essential Training
    • Up and Running with Git and GitHub
  • Books
    • Pro Git
    • Version Control by Example, online book
  • References
    • Git Reference
    • Visual Git Reference
    • Overview of Git Tutorials (many good ones)
    • Github documentation
    • GitHub Desktop

• Introducing git and GitHub
  • Content Addressable Filesystem and Snapshots
  • Distributed: Local vs. Remote development
  • .git directory
  • Content Integrity, SHAs (Secure Hash Algorithm)
    • git init
    • echo 'test content' | git hash-object -w --stdin
    • ls .git/objects
    • git cat-file -p d670460b4b4aece5915caf5c68d12f560a9fe3e4
  • Blobs, Trees, and Commits
  • Branches, master
  • Working Directory, Staging Area, Repository
  • What do these commands really do?
    • git clone url-to-git-repo
    • git add file.txt
    • git status
    • git rm file.txt
    • git commit -m "Added file.txt"
  • Remotes, origin, upstream

• Filing and Fixing a bug: a cookbook approach
  • set up git and GitHub
    • https://help.github.com/ has lots of great articles to help you. You can also view video guides or read the printed guides
    • setup your username in git
    • setup your email address in git
    • specify which editor git should use, for example you can use vscode
    • setup line endings (CRLF vs. LF) in git, extra notes for Windows users
    • setup ssh keys for GitHub
  • In GitHub, create a fork of the repo you want to work on
  • On your computer, clone your forked repo
  • On your computer, add a remote named "upstream" for the original repo (vs. your fork)
  • On GitHub, find or create an Issue for the change you want to make
  • On your computer, create and checkout a branch for your work, e.g., issue-1234 for Issue #1234
  • On your computer, make code changes, test them, add, and commit on your branch. Repeat as necessary.
  • On your computer, push your changes (commits) to your fork (origin)
  • On GitHub, create a Pull Request for your changes to get sent to the upstream repo
  • On your computer, fix any problems pointed out by your reviewer(s), add the file(s), commit, and push again to update your pull request

• Release 0.1 due Mon, Sept 23

Week 4

• Blogging
  • Why Blogging is Awesome
  • Keep reading the Planet, there are great posts there

• Learning Licenses: MIT
  • MIT License
  • The MIT License, Line by Line
  • One of the most widely used licenses in Open Source
  • Like the BSD License, nothing about patents (created before software was patentable in the US)
  • Example software projects licensed under the MIT License:
    • ExpressJS
    • Ruby on Rails
    • AngularJS
    • Atom, Electron
    • Bootstrap
    • node.js
    • Phaser
    • PuTTY
    • Socket.IO
  • "Do Good" forks of MIT:
    • JSON License with a great overview and history here.
    • Hippocratic License
    • Recent situation with Chef-Sugar, Chef and ICE Contract. Update from today

• More Git
  • Git Walkthrough Part I
  • Git Walkthrough Part II
  • Upstream and keeping your master branch up-to-date

• Release 0.1 due Mon, Sept 23. Any issues you need help with?
• Release 0.2
• Lab 3
• Startup Open House Event Tonight, Thurs Sept 26 4-8

Week 5

• Forking vs. Merging
  • Anyone can fork, not everyone can get work merged back in
  • My Repo: my house my rules
  • Some famous Forks
    • Firefox from Mozilla Suite
    • WebKit from KDE's KHTML
    • Blink from WebKit
    • Ubuntu from Debian
    • Sun's StarOffice became OpenOffice became LibreOffice
    • WordPress from Cafelog
    • MariaDB from MySQL
    • FireOS (Amazon for Kindle) from Android
    • io.js from node.js, which eventually became the official node.js
  • Example Fork
    • Semantic UI
    • Fomantic UI - "Fomantic was created to continue active development of Semantic-UI and has the intent to be merged back into the master repository once active development can restart."

• Merging with git
  • Where git branch splits histories apart, git merge brings them back together
  • Understanding DIFFs and Patch files
    • git diff, git show, git log -p, etc. to show DIFFs
    • Pull Requests also have links to get the raw .diff and .patch
    • How to read a DIFF file
  • Types of Merges: Fast Forward, Recursive Merges are the most common
    • --ff-only to force a fast-forward (only the branch pointer is moved, no new commit is created)
    • 3-way merges: two branch commits with a common ancestor (new commit is created with multiple parents)
    • Can have any number of parents though: one of the larges is a 66 commit octopus merge in the Linux kernel
  • How to merge
    • start with a clean working directory
      • commit your work if you can; or
      • stash (git stash list, git stash show, git stash pop)
    • checkout the branch you want to merge into
    • git merge branch_to_merge_into_this_branch
  • Various flags and commands to know:
    • git merge --squash
    • git merge --abort
    • git merge --continue
    • git branch -d
  • Merge Conflicts
    • Conflict markers <<<<<<<<<, =============, >>>>>>>>>>>>
  • Doing big merges in git

• TODO
  • Merging 3 PRs example
  • Release 0.2
  • Lab 4 - Complete your first Hacktoberfest PR, and write your first blog post about your fix. See guidelines in the Release 0.2 page. Place both links in a new section under the Submission section.

Week 6

• 0.2 Updates
  • How's it Going?
    • Things I read in your blog posts:
      • Setting up projects takes time, research (Docker, Gatsby, Makefile, Gems, npm builds etc)
      • Trouble running projects
      • Watch Windows line ending issues
      • Need to fix things the way the project wants vs. how you want
      • Include links in your blog posts when you talk about things
  • Interesting projects you've found?
  • Stats for Week 1
    • 44 PRs: +3,358/-941 lines of code across 178 files in 37 repositories
    • 19 have already been merged
    • 21 people didn't get Lab 1 done (no PR/blog)

• git commit --amend
  • Add to, correct, or otherwise alter your previous commit and/or comment message
  • Use --no-edit to leave the commit message alone

• git rebase branch
  • Replay commits on a new base branch/commit
  • Process goes like this:
    • git finds a common ancestor commit of the branch you're on, and the one you're rebasing onto
    • git calculates DIFFs for each, saves them to disk
    • git checks out the commit you want to branch onto, and begins to replay those diffs one by one
    • if there is a merge conflict, the rebase pauses so you can fix things
    • use git rebase --continue or git rebase --abort to move forward after such a pause
    • use git rebase --skip to ignore the current commit and keep going
  • Never rebase commits that are shared publicly in another repo. Only do it on commits you own locally (e.g., a topic branch you are working on)
  • Don't use rebase to get rid of commits in a public branch, use git revert commit-sha instead to apply an inverse commit
  • If you rebase a branch you've pushed (e.g., for a pull request), when you push, use git push origin branch-name -f (f means force and will overwrite)
  • git rebase -i for interactive rebase
    • shows a script of all commits in reverse order (order they will be replayed). You can hand edit this to remove, re-order, or combine commits
  • You can squash on the same branch by rebasing on HEAD~n where n is how many commits back from HEAD to go

• git cherry-pick SHA to add a commit to the current branch

Week 7

• Continue working on 0.2
  • Discussion of any issues/questions you have
  • End of Week 2 Stats:
    • +15,819/-8,733 LOC in 414 files across 116 PRs to 89 Repos. 62 merged (53%) already
  • Interesting stories from PR #2
    • Neil Ong
    • James Inkster
    • Hayley McIldoon
    • Krystyna Lopez and Julie James
    • Josue Quilon Barrios

• Open Source Case Study: Visual Studio Code
  • https://code.visualstudio.com/
  • https://github.com/Microsoft/vscode
  • https://en.wikipedia.org/wiki/Visual_Studio_Code
  • Technologies
    • Electron
    • Monaco Editor
    • TypeScript
    • xterm.js
    • node.js, express, and hundreds of JavaScript modules
  • Looking for Contribution Opportunities in Dependencies
    • yarn
    • Look at node_modules and examine the projects and issues (955 deps)
    • Are there any bugs we could fix?

• Fixing a Bug in VSCode Walkthrough:
  • https://github.com/humphd/vscode/tree/good-first-experience-issue-42726#walkthrough-fixing-a-bug-in-visual-studio-code

Week 8

• Continue working on 0.2
  • Discussion of any issues/questions you have
  • End of Week 3 Stats:
    • +30,402/-14,751 LOC in 688 files across 178 PRs to 127 Repos. 89 merged (50%) already
  • Interesting stories from PR #3
    • Ana Garcia
    • Miguel Roncancio
    • Minuk Park
    • Matthew Clifford

• Discussion of OSD & DPS909 Fall 2019 - Release 0.3 and OSD & DPS909 Fall 2019 - Release 0.4
  • Creating an Open Source Project as a group of 50+
  • Telescope project, see the project Overview doc
  • Seneca Open Source Slack for async communication (sign-up with your @myseneca.ca email)

• Wrap-up and Discussion of Hacktoberfest 2019

• Discussion of Labs for the rest of the term.
  • Every week you'll need to contribute in some way to our open source project. This could be a small PR, reviewing code, adding some docs, writing a test, working in issues, doing research and writing background material, etc.
  • Your lab blog post each week should discuss what you did

• Readings for this week: source code for Planet

Week 9

• Hacktoberfest 2019 Stats
  • 62 Students and a 95% completion rate (4 or more PRs)
  • 246 PRs, with 56% currently merged
  • 647 Commits to 881 files
  • 32K lines of code added, 14K lines removed
  • Languages
    • Most popular: JS/TS (124), Python (25), HTML/CSS (28), C/C++/C# (27), Java (14)
    • New/Interesting: Swift (4), PHP (3), Go (2), Rust (2), and 1 in each of OCaml, PowerShell, Ruby, Elixir, Kotlin, Shell
  • Notable Projects:
    • xterm.js, Polymer, Bitcoin, Angular, Ethereum, VSCode, Microsoft (Calculator, React Native Windows, STL), Jest, Facebook, WordPress, node.js, Nasa, Salesforce, Mozilla, Home Assistant, Google, Instacart,

• Releases 0.3 - due Friday Nov 15
  • Be working on your external PR
  • Everyone needs a telescope Issue by the end of the week. We need to make this happen together
  • Friday's Lab will require you to document what you are doing for both.

• Open Source Case Study: Redis
  • Redis (REmote DIctionary Server)
  • https://github.com/antirez/redis - ~175K lines of code
  • Cross-platform, high performance, in-memory, key/value, data structure database server. Written in mostly in C, as well as Tcl and Lua, with front-ends in just about every language and platform.
  • BSD 3-Clause
  • Started in 2009 by Salvatore Sanfilippo (antirez)
  • Since 2015, development has been sponsored by Redis Labs (see https://en.wikipedia.org/wiki/Redis_Labs)
  • Redis is among the most popular NoSQL databases in the world, and the most popular key/value store. It is used by everyone:
    • Twitter, Instagram, GitHub, StackOverflow, Pinterest, Snapchat, Shopify, Airbnb, Uber, Tumblr, Slack, Medium, Imgur, Kickstarter, etc.
  • Common Use Cases:
    • User Session Cache (e.g., reduce DB lookups for user info, shopping cart data)
    • Full Page Cache (e.g., by URL or route)
    • Queues (e.g., Message Queue, Worker Queue)
    • Counting (e.g., metrics, analytics)
    • Pub/Sub (e.g., chat systems, notifications)
  • Redis Tutorial and Walkthrough: https://try.redis.io/
• Writing Code Comments based on 32e0d237 commit
  • "[W]riting comments is of paramount importance in order to produce good code, that is maintainable in the long run and understandable by others and by the authors during modifications and debugging activities."
  • "So comments can be, for me, a tool for lowering the cognitive load of the reader."
• His YouTube channel has more video discussions of the code in Redis

• Redis, Bull, and telescope
  • https://github.com/Seneca-CDOT/telescope/pull/24
  • Writing a Twitter clone using Redis
  • Projects using Bull we can read and use as models to learn patterns:
    • https://github.com/sourcegraph/sourcegraph
    • https://github.com/coralproject/talk
    • https://github.com/zeit/node-file-trace
    • https://github.com/salesforce/refocus
    • https://github.com/OriginProtocol/origin
    • https://github.com/syuilo/misskey
    • https://github.com/Requarks/wiki
    • https://github.com/Chocobozzz/PeerTube
    • https://github.com/GetStream/Winds
    • https://github.com/withspectrum/spectrum
    • https://github.com/outline/outline
    • https://github.com/LearningLocker/learninglocker
    • https://github.com/ParabolInc/action
  • https://github.com/Seneca-CDOT/telescope/issues/112 has notes on some research into these projects. Please add more as you find it.

Week 10

• Managing change on a fast moving project with lots of people
  • Documentation doesn't work, project "rules" don't work
  • Communication works (Watching Issues, Slack, in person discussions) and
  • You have to enforce standards across a project using tools and automation
  • Continuous Integration (CI) - we have Travis CI enabled
  • Static Analysis Tools
    • we have eslint for JavaScript using the Airbnb JavaScript Style Guide
    • we have stylelint for CSS
  • Automated Tests with Jest
  • Next we will add Prettier for standard code formatting. Get your 0.3 PRs in before the weekend, or get ready for merge conflicts when this lands.

• Case Study - Prettier
  • Web site: https://prettier.io/
  • Twitter: https://twitter.com/PrettierCode
  • GitHub: https://github.com/prettier/prettier
  • James Long introducing Prettier (video)
  • Visualization of Prettier Development (video)
  • Installing Prettier

• Reviews
  • PRs must have a title that explains the fix, links to an Issue #
  • PRs must have a full description. If this is UI code, show a screenshot, or explain the fix, talk about how to review it, how to test it, what's not done, what's going to happen in further PRs, etc.
  • All PRs must pass CI: eslint, stylelint, unit tests
  • Don't let people land code with unrelated commits (e.g., merges with master)
  • Ask people to update their master with upstream, and rebase
  • Don't let people land code with changes to unrelated lines/files (e.g., package.json, whitespace changes)
  • Make sure one PR doesn't undo the work of another (e.g., bad merge, erasing existing code)
  • Ask yourself how we'll test every piece of code we take. We need to be able to trust it going forward.
  • PRs for us to try reviewing together:
    • #102 - Added Extracting of URLs from blog feed function
    • #123 - Fix #74: Validate blog URLs.
    • #136 - Added feed list parser
    • #137 - added pagination to feed for #40

• Research on Bull/Redis code
  • https://github.com/Seneca-CDOT/telescope/issues/112 has lots of ideas for Issues you could file and fix

Week 11

• 0.3 Recap
  • Make sure your PRs, Issues, and Blog posts are posted on the 0.3 page. I'll mark 0.3 and 0.4 together, in case you're still cleaning up work from the past week (you have time).
  • Over 100 Pull Requests, 67% are already merged
  • 476 commits to 325 files totaling more than 20K lines of code edited/added
  • Top Repos: Telescope! followed by Microsoft, Mozilla, Facebook, Uber, Zeit, Elastic, GatsbyJS, .NET, and 100 others

• Telescope Updates
  • Gource visualization of the repo
  • Tour of the Code
  • TODO - Plan for 0.4
  • How to get involved
    • triage issues
    • reviews
    • help debug things others are stuck on in PRs
    • help people with git issues
    • fixing bugs
    • project management

• Discussion of git Workflows
  • Value of CI - Travis CI builds and Circle CI builds
    • CI for Firefox
  • Rebasing vs Merging
  • Linear history is better than a tangled web of merges
    • gitk on Telescope vs.
    • gitk on Prettier, node.js, WebKit, Gutenberg
  • Many, Small PRS are better than a few big ones
    • easier to review
    • easier to merge
  • Break your work up, leverage the community of developers we have
  • Dealing with package.json and package-lock.json

• Case Study
  • Walkthrough: using history to debug with git bisect

Week 11

• 0.4 Status Check
  • Pull Requests
  • Issues
  • Countdown: 2 weeks to a working program. What risks do we see? How to address them?

• Case Study: Code Review at Google
  • Creating Reviewable Code
  • Preferring Small Changes
  • The Standard Code Review
  • How to Review Code
  • How to Handle Review Comments

• Case Study: Shopify developers using git, CI, merging
  • Over 1000 developers working on the code
  • Shopify changes 40 times a day
  • 400 commits a day to master
  • Master must always be green (passing CI)
  • Master must stay close to production
  • Emergency merges must be fast
  • /shipit GitHub integration
  • Merge queue to manage merges to master