Friday, November 4, 2016

TIP: recover file from old Git branch

In cleaning up my source code I realize I was too... aggresive in my house cleaning. I deleted a file I needed.  No biggie, just take a look at HEAD or HEAD^... Hey, that didn't work!

TIP: use git rev-list to search and find the most recent commit that affected (deleted) a path. (Source: Charles Bailey on SO)

$ git rev-list -n 1 HEAD -- docker/torta/torta-task.json
0c400d323aff484cfb2bdc18dcdb7813de93a658

Yay!  That was really easy. Now all I need to do is check out from that Git hash and I'm done:

$ git checkout 0c400d323aff484cfb2bdc18dcdb7813de93a658 !$
error: pathspec 'docker/torta/torta-task.json' did not match any file(s) known to git.

Err?  Hmm. Oh: the 0c4 branch contains the "delete" command. If I want to get the file contents, I'll get it from the branch just before that one:

$ git checkout 0c400d323aff484cfb2bdc18dcdb7813de93a658^ docker/torta/torta-task.json
$ ls -l docker/torta/torta-task.json
-rw-r--r--  1 johnm  staff  1445 Nov  4 13:06 docker/torta/torta-task.json

Yay, it worked!  Thanks, interwebs!

Tuesday, November 1, 2016

Videos! For "Platonic Solids of Quality" and "Functional Programming and Django QuerySets"

I've been remiss in posting my videos.  The last two talks went really well

- Platonic Solids of Quality

As people who produce awesome stuff, there are a lot of tools we can use to work less and simplify our workflow. This talk introduces many techniques from Agile Testing and other sources so we can all do more things faster.

- Functional Programming and Django QuerySets

This talk is a quick overview of Python iterators and why to use them, then combining iterators in Functional Programming (vs procedural and object oriented programming). Lastly I show how to use these ideas in order to better understand and test Django QuerySets.

I have another talk video somewhere, but it's about using Selenium for testing and thus has rather a bit of profanity in it.  No link.

Thanks to Don Westland for recording-editing these for me!

Friday, October 28, 2016

TIP: use Strace to debug issues inside Docker

Yesterday my Docker application wasn't working correctly -- the appserver is hanging.  Debugging this is a challenge: there's no crash nor stack trace to point out the issue.  Is the appserver misconfigured, so it's trying to talk to a non-existent database? Is the config okay, but the network is not set up correctly?  Can an external service not see our Docker container correctly?

To debug this I used my good old "strace" command to trace exactly what is happening. It outputs log messages for all system calls the appserver does, including all the network I/O.  Alas it didn't work for me:

strace: test_ptrace_setoptions_for_all: PTRACE_TRACEME doesn't work: Operation not permitted

This is odd, as the Docker container is running with root permissions, and the parent container is Debian.

My buddy Loren says this is a Docker thing -- the ptrace system call (which strace uses) is disabled by default.  To run a Docker container, re-enabling ptrace, run this:

docker run -i -t --security-opt=seccomp:unconfined --rm debian sh -c 'apt update ; apt install -y strace; strace -e trace=network ping -c1 8.8.8.8'

The above command does some Docker stuff, then sends a single ICMP ping packet to Google's global (and easily-remembered) DNS server.

Output:

socket(PF_INET, SOCK_RAW, IPPROTO_ICMP) = 3
setsockopt(3, SOL_SOCKET, SO_BROADCAST, [-68864700], 4) = 0
PING 8.8.8.8 (8.8.8.8): 56 data bytes
sendto(3, "\10\0\363\330\0008\0\0b\207\23X\0\0\0\0\4~\f\0\0\0\0\0\0\1\2\3\4\5\6\7"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = 64
recvfrom(3, "E\0\0TU\r\0\0%\1\204y\10\10\10\10\254\21\0\2\0\0\373\330\0008\0\0b\207\23X"..., 136, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, [16]) = 84
64 bytes from 8.8.8.8: icmp_seq=0 ttl=37 time=0.339 ms
--- 8.8.8.8 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.339/0.339/0.339/0.000 ms
+++ exited with 0 +++

Note the "sendto" and "recvfrom" system calls.  They show that the Docker container can talk to the internet.  This is verified with the "packets" line.

Strace is so awesome, Julia "b0rk" Evans wrote an entire zine about it! ~ http://jvns.ca/zines/

Thanks Loren and Julia!


Wednesday, October 26, 2016

Never type a Bash command again with Infinite Shell History!

To turn on infinite history, add this to your ~/.bash_profile file :

# infinite timestamped history
export HISTTIMEFORMAT='%F %T '
export HISTSIZE=''

When next you log in, your terminal history will be saved forever!  I'm up to about 30,000 commands typed in the last two years.

Example

Today I was looking up that command, you know, that one where you can hop in to Docker container running the main appserver, for debugging and testing.  IE: run a Bash shell on a named container.

I knew it was "docker exec..." something:

history | egrep docker.*exec

26790  2016-08-31 16:18:12 docker exec -it e3f bash

Oops, I found the command where I created a Bash shell in a specific Docker container.  I'd have to used "docker ps", found the specific ID, and typed a second command to run a shell.  Lame.

How about running a command that uses docker ps to list containers, and gave it a "filter", so that the same command always works?

history | egrep docker.*exec.*filter

24559  2016-08-05 14:07:37 docker exec -it $(docker ps --filter ancestor=theblacktux_web,status=running --quiet) bash

That's it!

The above command will create a Bash shell in the currently running Docker container, where the container is running a custom version of the "theblacktux_web" appserver image.

Wednesday, October 12, 2016

TIP: more info in Python's IPDB debugger

In ipdb the “dunder exception” variable has more information after your program crashes:

pdb> pp __exception__

(,
 ClientError...)

Tuesday, October 4, 2016

talk: Upping Your (Programming) Game

(this page is http://bit.ly/jta-upping)

I'm giving this talk tonight, at LA Django, hosted by our friends at Wiredrive HQ. Thanks for Marcel and Wiredrive for supporting the Los Angeles tech scene!

Slides are in gdrive: Upping Your (Programming) Game.

There will be cats Lil Bubzilla.


pic credit: Marcel


Wednesday, September 28, 2016

TIP: work with Amazon AWS ECS Container Service Docker containers

When running Amazon EC2 Container Service (ECS), have these two commands running in Tmux on your instances:

tail -f $(ls -1t /var/log/ecs/ecs-agent.log.* | head -1)

sudo tail -f /var/log/upstart/docker.log


In another Tmux tab, this command lets you look at the logs from your running Docker container:

docker logs -f $(docker ps --filter name=myname,status=running --quiet)