ARTS - 2019 Week 6-5¶
20190630~20190706
Algorithm¶
- 26. Remove Duplicates from Sorted Array
- 27. Remove Element
- 28. Implement strStr()
- 35. Search Insert Position
Review¶
TuneIn: How to Get Your Hadoop/Spark Jobs Tuned While You’re Sleeping¶
Have you ever tuned a Spark, Hive or Pig job? If yes, then you must know that it is a never ending cycle of executing the job, observing the running job, making sense out of hundreds of metrics and then re-running it with the better parameters. Imagine doing this for tens of thousands of jobs. Performance optimization at this scale manually is tedious, requires significant expertise and results into wasting a lot of resources to do the same task again and again.
As Hadoop/Spark is the natural choice for any data processing with many naive users, it becomes important to develop a tool to automatically tune Hadoop/Spark jobs. At LinkedIn we tried to solve the problem using Dr. Elephant, an open-sourced self-serve performance monitoring and tuning tool for Hadoop and Spark, used at LinkedIn and various other companies. While it has proven to be very successful, it relies on the developers’ initiative to check and apply the recommendation manually. It also expects some expertise from developers to arrive at the optimal configuration from the recommendations.
In this talk we will discuss TuneIn, an auto tuning framework developed on top of Dr. Elephant. We will describe how we are using an iterative approach of optimization to find optimal parameters. We will discuss the various optimization algorithms we tried and why we found Particle Swarm Optimization algorithm to give best results. We will talk about how we avoided using any extra execution and tuned the jobs during their regular scheduled execution. We go into detail on techniques employed to ensure faster convergence and zero failed executions while tuning. We will showcase how we achieved more than 50% gain in resource usage by tuning a small set of parameters. We will also talk about the lessons learned and the future roadmap.
【PDF】TuneIn: How to Get Your Hadoop/Spark Jobs Tuned While You’re Sleeping
背景¶
- 提升资源利用率
- 减少作业执行时间
Dr. Elephant¶
Particle Swarm Optimization (PSO)
- 搜索空间：待调优的参数
- 优化：避免失败、加快收敛、约束优化、参数依赖
- 损失函数：预估资源的使用
- 惩罚函数：避免过拟合，导致资源分配过于苛刻
流程
- 作业调度系统获取作业执行参数
- 返回优化后的参数
- 作业调度系统提交作业
- 获取作业度量信息，分析作业
特征
- 对周期运行任务调优，可以自动关闭调优
- 控制资源使用和执行时间，支持hive、spark，易于集成
- 容错处理，参数调整约束，失败处理
规划
- 优化执行时长
- 通过IPSO提升收敛速度
- 更智能的优化开关
Tip¶
Kafka 集群参数¶
Broker
- 日志存放路径：
log.dirs=/data/kafka1,/data/kafka2 - ZooKeeper 集群地址，协调、管理，保存 Kafka 集群元数据信息：
zookeeper.connect=zk1:2181,zk2:2181,zk3:2181/kafka1 - 监听器，访问 Kafka 服务的协议、主机名和端口：
listeners=PLAINTEXT://localhost:9092 - 不允许自动创建 Topic：
auto.create.topics.enable=false - 不允许 Unclean Leader 选举：
unclean.leader.election.enable=false - 不允许定期进行 Leader 选举：
auto.leader.rebalance.enable=false - 消息保存的最长时间：
log.retention.hour{minutes|ms}=168 - 总体消息保存的最大容量：
log.retention.bytes=100 000 000 000 - 接收每条消息的最大大小：
message.max.bytes=1 000 000
Topic
- 消息保存的最长时间：
retention.ms=604 800 000 - 接收每条消息的最大大小：
max.message.bytes=1 000 000
JVM
- 堆内存设置：
export KAFKA_HEAP_OPTS=--Xms6g --Xmx6g - GC参数设置：
export KAFKA_JVM_PERFORMANCE_OPTS=-server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35
OS
- 文件描述符限制：
ulimit -n 1000000 - 文件系统类型：
XFS、ZFS - 调整 Swap 空间比例：
vm.swappiness=1 - 关闭访问时间