Member since
11-22-2016
83
Posts
23
Kudos Received
13
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2068 | 08-03-2018 08:13 PM | |
1842 | 06-02-2018 05:24 PM | |
1248 | 05-31-2018 07:54 PM | |
2009 | 02-08-2018 12:38 AM | |
1434 | 02-07-2018 11:38 PM |
09-24-2018
04:28 PM
1 Kudo
@Maxim
Neaga
It is safe to ignore the error related to __AtlasUserProfile. Its a false positive.
... View more
08-03-2018
08:13 PM
1 Kudo
Entity: Representation of real-world element within Atlas.
Atlas will capture aspects of the element that will be relevant from
metadata perspective. Relationship: How entities are
related to each other. This relation enforces aspects like lifetime and
containment. Different types of relationships: Composition: If one is deleted, the other is deleted as well. E.g. Table and Columns. If table is deleted all the columns will be deleted too.
Aggregation:
If one is deleted other can continue to exist. E.g. Database and Table.
If a table within a database is deleted, database will continue to
exist. Relationships help sound modeling of data. Classification:
This is broad categorization of entities. Entities that are related
from a business perspective in some way are classified with same
classification. E.g. Sensitive information will reside in several tables
in several database in a data warehouse. A classification like
'Sensitive' can be applied to those tables.
Hope this helps!
... View more
07-02-2018
05:22 PM
Can you please post logs from /var/log/atlas/appplication.log, if this file is empty, please see contents of *.err, start-up errors because of resource constraints, they will be logged in the .err file.
... View more
06-02-2018
05:24 PM
Your question is valid. In most cases the ways of 'knowing' existence of entity are redundant. In general, GUIDs are assigned to entities when they are created, this remains unchanged through the lifetime of the entity. In case of entity creation via hooks: The incoming
entities will not have GUIDs as they are yet to be created, however
qualified name is available as it is required attribute for entities. This is used to detect existence of entities. Qualified name has potential for change in the this scenario: Entities are moved across cluster as part of synchronization using Export & Import APIs. The GUID main same but qualified name can change to reflect the correct location of the entity. E.g. Database entity within cluster with name cl1 will have entities whose name is db@cl1. Importing this entity into a cluster with name cl2 should change the qualified name to db@cl2. This way the imported entities reflect their new home, whereas with guid it is possible to know that the same entity exists across clusters. Please take a look here in transforms. Hope this helps.
... View more
05-31-2018
07:54 PM
Your observation is valid. The version number is something related to the model (schema, definition is present in addons/models) of the entity and not the version of the entity itself. Also, as of now, we don't have logic for dealing with version changes to schema. In short, version field is not of much consequence right now.
... View more
05-31-2018
04:04 AM
Please take a look at these models that we recently added.
... View more
03-18-2018
06:25 PM
Do you see any exceptions in the logs?
... View more
02-20-2018
04:06 AM
Introduction This post enumerates the steps necessary to setup Atlas development environment using IntelliJ on Mac and Windows. This setup uses the BerkeleyDB for backend and embedded Solr as index engine. Setup with other backend and index engine variations are similar but will involve additional setup. Prerequisites These should be present on your machine before you begin:
Git for cloning repository. The command Git Shell is useful if you are switching between Mac and Windows. Maven for performing command line build. IntelliJ Community Edition or higher. BerkeleyDB as backend. Code Base Setup Download code base from GitHub location. Clone it under c:\work\Apache\atlas on Window and ~/Apache/atlas on Mac. Change directory to the location above and initiate a build (using mvn clean install package). Deploy Directory Setup Create a directory say Deploy (say c:\work on Windows or ~/work on Mac) with a structure below it:
conf
Copy atlas-application.properties, users-credentials.properties, policy-store.txt atlas-log4j.xml and atlas-env.sh here. Use contents of the attached ZIP. data
During runtime, the backend database will create its files here. This may be a location to check. data/solr Copy contents C:\work\atlas\repository\src\test\resources\solr to c:\deploy\data\ for Windows (from ~/Apache/atlas/repository/src/test/resources to ~/Deploy/data on Mac). libext
Copy BerkeleyDB JAR here, say je-5.0.73.jar logs
Logs will be created here. models
Copy contents from c:\work\atlas\addons\models (or ~/Apache/atlas/addons/models for Mac.) webapp (optional)
Deploy the contents of atlas.war here if you are developing on client-side (UI). bin (optional)
Empty for now. When done, your directory layout should look like this: WinUtils (for Windows only) Install WinUtils (link below). Copy WinUtils.exe from C:\Program Files (x86)\WinUtil\WinUtil.exe to C:\Users\ashut\.m2\repository\org\apache\hadoop\bin\WinUtils.exe IntelliJ: 'Atlas - Local' Configuration From IntelliJ's Run/Edit Configurations menu option, create a new configuration, call it 'Atlas - Local'. Details are:
Type: Application Main class: org.apache.atlas.Atlas VM options: These should reflect location of your directory created in the step above. Add the following:
-Datlas.home=C:\work\deploy\ -Datlas.conf=C:\work\deploy\conf -Datlas.data=C:\work\deploy\data -Datlas.log.dir=C:\work\deploy\logs -Dembedded.solr.directory=C:\work\deploy\data (See screen shot Profile-2) Program arguments: --port 31000
This is needed so that Atlas that is being run from IntelliJ does not clash with another version that runs on the development VM. Working directory: Set this to the location of webapp of your code base. (In my case, c:\work\apache\atlas on Windows and ~/Apache/atlas/ on Mac) Use classpath of module: atlas-webapp See screen shots below. Debug Run Within IntelliJ: Set the newly created configuration as active. From View/Tool Windows/Maven Project enable the Maven Projects side pane. From the Profiles select Berkeley-elasticsearch, graph-provider-default, graph-provider-janus. Use Run/Debug - 'Atlas - Local' from the menu. Check if server is up by accessing: http://localhost:31000/ Screen Shots Atlas - Local Profile:
Attachments conf-directory.zip: Contents of configuration directory. References How to install Maven on Windows WinUtil download. Credits
Thanks to Apoorv Naik (@anaik) for the investigation and the coming up with setup steps and helping me with the many setups.
... View more
Labels:
02-08-2018
12:38 AM
1 Kudo
(@Sarath Subramanian Thanks for your help on this!) Version number is primarily used for patch updates to existing data types. On master, note the patches directory under addons/models/1000-Hadoop/patches AtlasTypeDefSoreInitializer.loadBootstrapTypeDefs (repository/store/bootstrap/AtlasTypeDefStoreInitializer.java) scans this directory and updates type definitions based on the version. In short, it is used for maintenance of types. Beyond this, version is NOT used internally for validation, etc.
... View more
02-07-2018
11:38 PM
Thank you for your patience. Attached is a sample from one of the internal environments. The attached zip (tag0.zip) has 2 files:
tag0.json: Top-level tag containing 1000+ sub-types. tag0_207.json: One of the child tags that do not have sub-types, but has tag0 as parent. You can use entity creation APIs to add these types to the Atlas server. Bulk APIs can be used to add multiple type definitions at the same time. Note that the bulk APIs takes AtlasTypesDef as input, ensure that the posted JSON is in that format. curl -X POST -u admin:admin -H 'Content-Type: application/json' -d @tags.json http://localhost:21000/api/atlas/v2/types/typedefs
Hope this helps.
... View more