Coding - Operation Systems


  • Copy file to another server

scp prime.war lchen@ny-sample-002:/home/lchen

  • Grep log with regular expression from the zip file:

zgrep –color ‘CCVI[0-9]{12}’ /logs/pl_timings.17-11-01_21.log.gz

  • Check if a domain name accessible from DNS

dig +noall +answer	1799	IN	CNAME	3600	IN	CNAME 30	IN	A
  • Curl with proxy and client certificate

curl -E “./Priceline DB.pem” –key “./Priceline DB.key” -H “Authorization: orgId=116770” –proxy “”

  • Create a pkcs12 (.pfx or .p12) from OpenSSL files (.pem, .cer, .crt,…)

openssl pkcs12 -export -in .pem -inkey .key -out .p12

  • Install certificate to JVM keystore

sudo keytool -import -alias sunas -keystore /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/lib/security/cacerts -file /Users/lchen/Downloads/

the keystore password is changeit by default

Install Hadoop in Ubuntu

sudo apt-get install default-jdk

Mac OS

  • If install Hadoop on Mac OSX, make sure turn on Remote Login under System Preferences then File Sharing.
  • Show the using TCP ports: sudo lsof -iTCP -sTCP:LISTEN -n -P
  • Show the latest changed files ls -ltr

MacOS VirtualBox

  • Change settings (advanced) to allow bidirectional copy & paste.
  • Install Guest Additions
  • Change Host Key Combination to right right command
  • Change Host Network manager, add a host-only network vboxnet0
  • Change Client Network Adapter2, choose Host-only Adapter
  • In Ubuntu, install/run openssh: sudo apt-get install openssh-server
  • In Ubuntu, check ip address with ifconfig -a
  • In Host terminal, you can now ssh to client, say ssh lchen@

Install Hadoop & Hbase on macOS

  • Prepare local SSH access
$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
$ cat ~/.ssh/ >> ~/.ssh/authorized_keys
$ chmod 0600 ~/.ssh/authorized_keys
$ ssh localhost
$ exit
  • Install Ruby & Brew
$ rvm install ruby-2.5
$ brew search hadoop
  • Install Hadoop
$ brew install hadoop
$ ls /usr/local/Cellar/hadoop
  • Update /usr/local/Cellar/hadoop/3.1.1/libexec/etc/hadoop/
export HADOOP_OPTS=""
  • Update /usr/local/Cellar/hadoop/3.1.1/libexec/etc/hadoop/core-site.xml
        <description>A base for other temporary directories.</description>
  • Update /usr/local/Cellar/hadoop/3.1.1/libexec/etc/hadoop/mapred-site.xml

  • Update /usr/local/Cellar/hadoop/3.1.1/libexec/etc/hadoop/hdfs-site.xml

  • Format HDFS and start all
$ hdfs namenode -format
$ sh

NameNode: http://localhost:9870/ Scheduler: http://localhost:8088/cluster/nodes

  • Install HBase
$ brew install hbase
  • Update /usr/local/Cellar/hbase/1.2.8/libexec/conf/hbase-site.xml