发表于 |

kafak 简单安装

  • 启动ZooKeeper 服务

    bin/zookeeper-server-start.sh config/zookeeper.properties

  • 启动kafaka服务

    bin/kafka-server-start.sh config/server.properties

  • 创建topic 名字叫:test

    bin/kafka-topics.sh –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic test

使用如下命令可以列出topic

bin/kafka-topics.sh –list –zookeeper localhost:2181
test

  • 插入几条 命令

    bin/kafka-console-producer.sh –broker-list localhost:9092 –topic test
    直接输入 ，从标准输入读取文件到kafaka

  • 读取kafak中的内容

    bin/kafka-console-consumer.sh –bootstrap-server localhost:9092 –topic test –from-beginning

发表于 |

linux 使用juniper ssl vpn

  1. 下载ncLinuxApp.jar 软件包

在浏览器输入https://xxxxxx/dana-cached/nc/ncLinuxApp.jar下载这个jar文件（xxxxx是你公司的vpn地址）

  1. 创建目录 并解压相关软件包

$ mkdir -p ~/.juniper_networks/network_connect/
$ unzip ncLinuxApp.jar -d ~/.juniper_networks/network_connect/
$ sudo chown root:root ~/.juniper_networks/network_connect/ncsvc
$ sudo chmod 6711 ~/.juniper_networks/network_connect/ncsvc
$ chmod 744 ~/.juniper_networks/network_connect/ncdiag

  1. 下载jnc软件包，复制到 /usr/local/bin目录下面 并且赋予执行权限

wget http://www.scc.kit.edu/scc/net/juniper-vpn/linux/jnc -o /usr/local/bin/jnc

chmo +x /usr/local/bin/jnc

  1. 安装32位兼容软件包
    On centos 6/7 and higher:

    yum install glibc.i686 zlib.i686 nss.i686

  2. 获取认证密钥
    cd ~/.juniper_networks/network_connect/
    sh getx509certificate.sh sslvpn.XXXXX.com 509.pem

  3. 创建配置文件
    mkdir -p ~/.juniper_networks/network_connect/config
    vi ~/.juniper_networks/network_connect/config/somename.conf
    host=sslvpn.XXXXX.com
    user=xxxx
    password=xxxx
    realm=FiberHome-realm
    cafile=/root/.juniper_networks/network_connect/509.pem

注意：realm 每个公司不一样的，请访问sslvpn.XXXXX.com 查看网页源码


其中value 为realm的值

  1. 启动/关闭
    $ jnc –nox somename

$ jnc stop

For more options see

$ jnc –help

启动成功空 可以ping 一下
10.19.8.10

  1. 排错
    ncsvc这32位程序 经常会有问题，请使用如下方法检测

[root@fonsview network_connect]# ldd ncsvc
linux-gate.so.1 => (0xf77b7000)
libdl.so.2 => /lib/libdl.so.2 (0xf778d000)
libz.so.1 => /lib/libz.so.1 (0xf7776000)
libpthread.so.0 => /lib/libpthread.so.0 (0xf775b000)
libm.so.6 => /lib/libm.so.6 (0xf7718000)
libc.so.6 => /lib/libc.so.6 (0xf7559000)
/lib/ld-linux.so.2 (0xf77b8000)

发表于 |

  1. 如果输入的密码过于简单，报错密码验证不通过可以
    Your password does not satisfy the current policy requirements
    [root@db1 ~]#vim ~/my.cnf
    validate-password = off

  2. Warning: Using a password on the command line interface can be insecure.
    比如：
    如下:
    [root@db1 ~]# mysql -uappuser -pappuser -e “show databases;”
    mysql: [Warning] Using a password on the command line interface can be insecure.

解决办法1：
[root@db1 ~]#mysql_config_editor set –user=root –password
Enter password:

上面文件会创建如下加密文件
[root@db1 ~]# cat ~/.mylogin.cnf

��u�q:������eƃ��>�C�R��U ��:=:M5&3��7�-<��jU�Q �uK�P�
[root@db1 ~]# mysql_config_editor print –all
[client]
user = root
password = *

这时，输入mysql命令时可不输入密码直接执行：

[root@db1 ~]# mysql -e “show databases;”
+——————–+
| Database |
+——————–+
| information_schema |
| mysql |
| ossdb |
| performance_schema |
| sys |
解决办法2：
创建明文:
[root@monkey-2 ~]# cat ~/.my.cnf
[client]
host= localhost
user = root
password = 1234

结果同上:

mysql_config_editor set –login-path=remote –host=remote –user=remote –password

[vagrant@localhost ~]$ mysql_config_editor set –login-path=remote –host=remote –user=remote –password
Enter password:secure
[vagrant@localhost ~]$ mysql_config_editor print –all
[client]
user = root
password = *
[remote]
user = remote
password = *
host = remote
[vagrant@localhost ~]$ mysql –login-path=remote

发表于 |

#oss简单检测命令：

#curl -v 127.0.0.1:8081/oss/

#curl -v 127.0.0.1:8081/oss/login.seam

发表于 |

sniffer

Sniffer

  • wireshark - Wireshark is a free and open-source packet analyzer. It is used for network troubleshooting, analysis, software and communications protocol development, and education. Wireshark is very similar to tcpdump, but has a graphical front-end, plus some integrated sorting and filtering options.
  • netsniff-ng - netsniff-ng is a free Linux networking toolkit, a Swiss army knife for your daily Linux network plumbing if you will. Its gain of performance is reached by zero-copy mechanisms, so that on packet reception and transmission the kernel does not need to copy packets from kernel space to user space and vice versa.
  • Live HTTP headers - Live HTTP headers is a free chrome extension to see your browser requests in real time. It shows the entire headers of the requests and can be used to find the security loopholes in implementations.

Monitoring / Logging

  • snyk.io - Snyk is a opensource tool to scan for vulnerabilities. Find, fix and monitor for known vulnerabilities in Node.js npm package fixtures used for testing the patches of vulndb.

  • justniffer - Justniffer is a network protocol analyzer that captures network traffic and produces logs in a customized way, can emulate Apache web server log files, track response times and extract all “intercepted” files from the HTTP traffic.

  • httpry - httpry is a specialized packet sniffer designed for displaying and logging HTTP traffic. It is not intended to perform analysis itself, but to capture, parse, and log the traffic for later analysis. It can be run in real-time displaying the traffic as it is parsed, or as a daemon process that logs to an output file. It is written to be as lightweight and flexible as possible, so that it can be easily adaptable to different applications.

  • ngrep - ngrep strives to provide most of GNU grep’s common features, applying them to the network layer. ngrep is a pcap-aware tool that will allow you to specify extended regular or hexadecimal expressions to match against data payloads of packets. It currently recognizes IPv4/6, TCP, UDP, ICMPv4/6, IGMP and Raw across Ethernet, PPP, SLIP, FDDI, Token Ring and null interfaces, and understands BPF filter logic in the same fashion as more common packet sniffing tools, such as tcpdump and snoop.

  • passivedns - A tool to collect DNS records passively to aid Incident handling, Network Security Monitoring (NSM) and general digital forensics. PassiveDNS sniffs traffic from an interface or reads a pcap-file and outputs the DNS-server answers to a log file. PassiveDNS can cache/aggregate duplicate DNS answers in-memory, limiting the amount of data in the logfile without loosing the essens in the DNS answer.

  • sagan - Sagan uses a ‘Snort like’ engine and rules to analyze logs (syslog/event log/snmptrap/netflow/etc).

  • OSSEC - OSSEC is an Open Source Host-based Intrusion Detection System that performs log analysis, file integrity checking, policy monitoring, rootkit detection, real-time alerting and active response. It runs on most operating systems, including Linux, MacOS, Solaris, HP-UX, AIX and Windows.

  • ntopng - Ntopng is a network traffic probe that shows the network usage, similar to what the popular top Unix command does.

  • Fibratus - Fibratus is a tool for exploration and tracing of the Windows kernel. It is able to capture the most of the Windows kernel activity - process/thread creation and termination, file system I/O, registry, network activity, DLL loading/unloading and much more. Fibratus has a very simple CLI which encapsulates the machinery to start the kernel event stream collector, set kernel event filters or run the lightweight Python modules called filaments.

发表于 |

stsc 统计系统环境

  • 整体结构图 

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
            +-------+
            | ss log|
            +---+---+
                | ftp up
            +---v---+
            |  omc  |v
            +---+---+                  +---+            +----+
                |  ftp down            |cms |           |oss |
                |  viewlog             +---+            +---++v
            +---v---+                    |   日志加工转换    |
            | flume |                    |    +-------+    |
            +---+---+                    +--->+ stdm  +<---+
     把viewlog  |                              +-+-+---+
     传给topic， | cdn-viewlog-pc                ^ |
                v                               | |
    +-----------+------------+    读,            | |
    |                        | -----------------+  |
    |      kafaka            | <------------------+
    +---------+--------------+    写，cdn-viewlog-pc-stsc
              |
              |读 cdn-viewlog-pc-stsc
              |
        +-----v-------+
        | fsvlogstash |
        +-----+-------+
              | 9300端口
        +-----v-------+    +-------------+    +-------------+
        |elasticsearch|<-->|elasticsearch|<-->|elasticsearch|--。。。。。。
        +-----+-------+    +-------------+    +-------------+
              |
        +-----v-----+
        |   stsc    |
        +-----+-----+
              |
        +-----v------+
        |stsc reports|
        +-----+------+
              |
          鉴权 | 单点登录
    +---+   +--v--+     +---+
    |oss+---+ cas +---->+cms|
    +---+   +---+-+     +---+
              | 
              |
    +---------v----------+
    |   喜闻乐见的图形界面   |
    +--------------------+

  • stsc 主要流程

  1. ss 生成log，并通过ftp的方式传送到omc

  2. 把omc中的log，下载到flume主机上

  3. flume解析 ss的log之后，把log内容传送到kafka里指定的topic中，比如 cdn-viewlog-pc

  4. stdm 从 kafka 中读取 未加工过的log，比如： 读取 cdn-viewlog-pc

  5. stdm 根据同步到的cms oss中的数据，和 cdn-viewlog-pc中的log，生成转换后数据。

  6. stdm 把转换后数据，传送到 kafka 里的 cdn-viewlog-pc-stsc 中。 注：转换后数据有后缀 stsc

  7. fsvlogstash 读取 cdn-viewlog-pc-stsc 中的内容 存入 elasticsearch 数据库集群中。

  8. stsc 提供api接口查询 elasticsearch

  9. stsc reports 调用stsc的接口，获得 elasticsearch中的数据，然后用图表呈现

  10. 登录 stsc reports的过程中，会跳转到cas中进行鉴权，cas的账户数据来源 cms、oss、本地cas数据库

  11. 经过cas的鉴权，就可以看到喜闻乐见的 stsc的统计页面了 。

发表于 |

/opt/fonsview/3RD/jdk1.8/bin/java -Djava.util.logging.config.file=/opt/fonsview/3RD/tomcat7.0.63/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djgroups.bind_addr=127.0.0.1 -Djava.net.preferIPv4Stack=true -server -Xms1000m -Xmx1000m -Xmn875m -Xss256k -XX:SurvivorRatio=6 -XX:MaxPermSize=1024m -XX:ParallelGCThreads=1 -XX:MaxTenuringThreshold=0 -XX:+UseConcMarkSweepGC -Djava.endorsed.dirs=/opt/fonsview/3RD/tomcat7.0.63/endorsed -classpath /opt/fonsview/3RD/tomcat7.0.63/bin/bootstrap.jar:/opt/fonsview/3RD/tomcat7.0.63/bin/tomcat-juli.jar -Dcatalina.base=/opt/fonsview/3RD/tomcat7.0.63 -Dcatalina.home=/opt/fonsview/3RD/tomcat7.0.63 -Djava.io.tmpdir=/opt/fonsview/3RD/tomcat7.0.63/temp org.apache.catalina.startup.Bootstrap start

tomcat 启动不起来很可能式内存不够

发表于 |

python 虚拟环境

  1. 安装
    pip install virtualenv

  2. 创建
    virtualenv –no-site-packages venv

  3. 使用虚拟环境

  4. 推出虚拟环境
    deactivate

  5. 单个软件包安装

pip install ansible -i http://mirrors.aliyun.com/pypi/simple/ –trusted-host mirrors.aliyun.com

发表于 |

php56 安装

yum –enablerepo=remi,remi-php56 install php php-mysql php-gd php-bcmath php-common php-mbstring php-xml –nogpgcheck