Work

Ambari

Docs

Avro

Docs

Docker

Docs

Eclipse

Docs

Flume

Docs

GitLab

Docs

Hadoop

Docs
Issues

  • java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

    Fix: add winutils.exe to ${HADOOP_HOME}/bin/ or add System.setProperty("hadoop.home.dir", "winutils.exe") code;

    Problems running Hadoop on Windows

  • java.lang.RuntimeException: Error while running command to get file permissions

    Fix: Set permissions: %HADOOP_HOME%\bin\winutils.exe chmod 777 \tmp\hive

  • org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4

    Fix: Hadoop package version conflicts, adjusting load order, or exclusion conflicts.

  • java.lang.UnsupportedOperationException: Not implemented by the DistributedFileSystem FileSystem implementation

    Fix: hadoop-core conflicts withDistributedFileSystem in hadoop-hdfs, adjusts the load order, orexclusion conflicts.

  • Job Stuck in ACCEPTED State.

    Fix: Adjust the configuration of the YARN Container or the YARN Scheduler.

  • No such file or directory. ／ RuntimeException Cannot make directory.

    Fix: Permissions problem.

  • java.lang.OutOfMemoryError. & Error: Java heap space. / Container [xxx] is running beyond physical memory limits.

    Fix: 

    1
    2
    3
    4
    5
    6
    # Maper
    mapred.map.child.java.opts=-Xmx4096m
    mapreduce.map.memory.mb=4608
    # Reducer
    mapred.reduce.child.java.opts=-Xmx4096m
    mapreduce.reduce.memory.mb=4608

  • Halting due to Out Of Memory Error

    Fix: 

    1
    2
    3
    # MRAppMaster
    yarn.app.mapreduce.am.command-opts=-Xmx2048m
    yarn.app.mapreduce.am.resource.mb=2560

HBase

Docs

Hive

Docs

IDEA

Docs

Java

Docs
Issues
  • Default Charset of the JVM(1.8)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
private static volatile Charset defaultCharset;

/**
 * Returns the default charset of this Java virtual machine.
 *
 * <p> The default charset is determined during virtual-machine startup and
 * typically depends upon the locale and charset of the underlying
 * operating system.
 *
 * @return  A charset object for the default charset
 *
 * @since 1.5
 */
public static Charset defaultCharset() {
    if (defaultCharset == null) {
        synchronized (Charset.class) {
            String csn = AccessController.doPrivileged(
                new GetPropertyAction("file.encoding"));
            Charset cs = lookup(csn);
            if (cs != null)
                defaultCharset = cs;
            else
                defaultCharset = forName("UTF-8");
        }
    }
    return defaultCharset;
}

System < Runtime < Cache < Code

Jenkins

Docs

Kubernetes

Linux

Maven

Mysql

Neuxs

Oozie

Docs
Issues

  • org.apache.oozie.action.hadoop.LauncherException: Output data exceeds its limit [2048]

    Fix: oozie.action.max.output.data=4096 # Max size in characters for output data.

  • Oozie Action Heart Beat (YARN Deadlock)

    Fix: Adjust the configuration of the Oozie Launcher or the YARN Scheduler.

OnlyOffice

Docs
Issues

  • 400 Bad request. File editing start error

    Fix: add parameter editingAlone=true.

Portworx

Docs

Rancher

Docs
Resources

Ranger

Docs
Issues

  • java.security.InvalidKeyException: Illegal key size

    Fix: obtain the JCE policy file appropriate for the JDK version in your cluster.

Spark

Docs
Issues

  • UTF-8 encoding data garbled in Spark job.

    Fix: Set runtime encoding: -Dfile.encoding=utf-8.

  • Error:(34, 45) object SqlBaseParser is not a member of package org.apache.spark.sql.catalyst.parser import org.apache.spark.sql.catalyst.parser.SqlBaseParser._

    Fix: Generated sources and make target/generated-source as a source folder.

  • Error:(45, 66) not found: type SparkFlumeProtocol val transactionTimeout: Int, val backOffInterval: Int) extends SparkFlumeProtocol with Logging

    Fix: Generated sources and make target/generated-source as a source folder.

  • NoSuchMethodError: scala.collection.immutable.HashSet$.empty()Lscala/collection/immutable/HashSet;

    Fix: Change scala-library version.

  • An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program “bash”

    Fix: Add Git’s bin directory to Path.

  • value toDF is not a member of org.apache.spark.rdd.RDD

    Fix: import spark.sqlContext.implicits._ && Move case class outside of the method

Spring

Docs

Storm

Docs
Issues

  • java.lang.ClassCastException: java.lang.String cannot be cast to [B or [B cannot be cast to java.lang.String

    Fix: //spoutConfig.scheme = new SchemeAsMultiScheme(new StringScheme());