FAQ

There are two different versions of DataFacet for SharePoint: DataFacet for SharePoint 2010 Server and Enterprise, and DataFacet for SharePoint 2010 Foundation Server and SharePoint 2007.

DataFacet for SharePoint 2010 is fully integrated with SharePoint and requires the Enterprise Keywords and either SharePoint Enterprise Search or FAST for SharePoint 2010 to be installed.

FAST for SharePoint requires SharePoint 2010 Enterprise edition.

Another options for search engine is DataFacet RebelSearch, which provides a high-performance search engine with complete deep facet navigation that is native to SharePoint, yet has a small footprint and is easy to configure.

For example: DEV, PROD, and TEST and the subsequent synchronization of terms across the various environments?

You can export DataFacet taxonomies to the industry-standard SKOS format and import them into another SharePoint farm.

We have a user story for a more automated process, but for now, this approach can be automated with PowerShell.

For example:  a term that has been edited or a rule added; is this audited and accessible from the Central Admin Server App user interface?

We do have logging, but it has not been designed specifically for auditing purposes. Better logging and auditing support is planned for a future release.

The rules are stored as properties on the term store objects. Ultimately they are stored in the SharePoint database, but this is an opaque data store.

Yes – but please don’t. DataFacet will require at least one IIS reset before going live. Plan on about an hour of down-time to upgrade to a new version of DataFacet on a production server.

Yes. DataFacet will not operate without a working Managed Metadata Service. You will need to configure an Enterprise Keywords service application.

DataFacet for SharePoint 2010 takes full advantage of the new Managed Metadata feature, which is not available in SharePoint 2007. Because of this, the architecture is fundamentally different, and DataFacet for SharePoint 2007 does not support tagging of documents stored in SharePoint Document Libraries.
We do support an interim solution for customers that are running SharePoint 2007 but are on an upgrade path to SharePoint 2010.

DataFacet Foundation with RebelSearch is our own search engine based on the popular Lucene search libraries.
Advantages include blazing fast performance, support for deep facet navigation, excellent integration with SharePoint and good integration with SharePoint 2007 and low cost relative to other options.

In addition, we support several other Enterprise Search applications if you happen to have them already, including Coveo, Exalead, Solr and MarkLogic. There is some ramp-up time associated with all of these solutions. These can be sensible alternatives for an organization that has already standardized on a third-party enterprise search engine.

Currently, all Windows-based AR and Datafacet products require at least Windows .Net framework version 3.5 Service Pack 1.
DataFacet for SharePoint has the same minimum requirements as SharePoint.

Additional Information: For example, we have a three farm 2010 configuration where there is a document collaboration farm, a social farm, and then a shared application service farm. Where would DataFacet need to be installed in this case?

Answer: When you install DataFacet on a Farm, you need to run the installer on every machine on the farm. The installer will automatically install the appropriate features for the topology of the server. So, if a server is a WFE without shared application features for the topology of the server. So, if a server is a WFE without shared application services, only the DataFacet UI components will be installed (templates, features). The DataFacet taxonomy service will be installed with other shared application services.

In order to successfully install DataFacet you will need to run it under account that has:

1. Local Admin Rights. We are copying files on the local file system & adding assemblies into the GAC.
2. SharePoint Farm Admin rights (i.e. account with write permissions to Config DB). We are installing sharepoint features, registering services, provisioning service application and more.
3. Term Store admin rights. We are adding term group called “Data Facet” at installation time.

Not yet. We are looking into options for this.

Our general Business taxonomy is available as a free download and is provided in CSV format.

Please visit http://www.datafacet.com/ for details.

The XML version (in encrypted SKOS format) of the DataFacet General Business taxonomy is shipped with DataFacet. You can find it in the 14 hive, commonly:

C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\WebServices\TaxonomyService\StaticData\DataFacet_GeneralBusiness_SKOS.tax"

This role is usually overseen by content administrators or similar business roles. IT roles and SharePoint architects are not necessarily the best choices for this task. Someone who is in charge of Master Data Management can typically transfer that knowledge to creating and managing taxonomies. In addition, company librarians can do an excellent job managing taxonomies; however, they are typically only present in large corporate environments.

Depending on the size of the company, anywhere from 4-8 hours a week. For larger companies this is often a full-time job and can even have multiple people engaged in managing the taxonomies.

ASPX documents are the same as HTML Documents. DataFacet does not process them any different from any other content type – they are handled by the protocol handlers and iFilters in SharePoint.

OWL is not supported natively, but it is very easy to convert from OWL to SKOS. We do support SKOS natively. We can help you with that. Just ask.

Since the Term Store only imports .CSV files, and special characters—even though they can be entered and used in the .CSV file—do not import correctly into the Term Store. Both UTF-7 and UTF-8 code produces the same result: special characters do not import correctly into the Term Store. For special characters have had to adjust/edit in the Term Store as necessary.
One work around is to identify terms with special characters in advance within the .CSV, notice any duplications, and copy or reuse those terms once they’ve been adjusted within the Term Store. This way, the term with the special character only has to be edited once and then copied where necessary, rather than importing the same term many times and having to edit them all within the Term Store. Not an elegant solution, for sure, but one that is a shortcoming of the Term Store.
Also DataFacet for SharePoint does handle UTF8 encoded XML files, so there is no problem importing taxonomy files in any of the XML formats that we support (e.g., SKOS and ARTX).

Yes, taxonomies and term store operates the same way in SharePoint Online as it does in SharePoint 2010. DataFacet Automatic Tagging Engine is currently not compatible with SharePoint Online however.

SharePoint 2010 itself does not support importing taxonomies in XML format. It only supports a specific variant of CSV. However, if you have taxonomies in XML which you wish to import into SharePoint, we can assist with transforming the data into the appropriate format on a services basis.

See http://technet.microsoft.com/en-us/library/ee424396.aspx for the SharePoint CSV import format.

DataFacet supports two XML schemas for import:

• SKOS – Simple Knowledge Organization System http://www.w3.org/2009/08/skos-reference/skos.html
• ARTX – Applied Relevance TaXonomy. This is a proprietary format that is easy to generate from hierarchical data.

DataFacet also supports a limited subset of RDF/XML documents with a constantly evolving fidelity.

SharePoint Foundation does not have a term store. You must be using SharePoint 2010 Standard or Enterprise

This would require either custom development through an API or a static import based on an export from your current system.

There are two distinct events.One is On Document Update, the other is On Term Store Update.

For the On Document Update event, we add a delegate to the built-in SharePoint API et voila, all documents checked into the library are annotated.

The on Term Store Update event is a different story. Since each document must be read in order to annotate it, it is an I/O intensive process, so it is inherently best done in batch mode in the background. Currently, we have a PowerShell script that can be scheduled to re-annotate documents in a given library.

DataFacet integrates well with records management systems out of the box by virtue of the support for the SharePoint Managed Keywords feature. We do provide audit logs through the normal SharePoint logging system, but there is no specific integration with records management at this time.

The terms get written to the index of the document that is stored in SharePoint but it does not change the contents of the document itself. The original document is never updated, only the SharePoint metadata.

We prefer FAST for a variety of reasons, not just the crawling capabilities. Primary for us on the technical side is the ability to have use the “deep facet” navigation that is available in SharePoint search but is not available in standard SharePoint search. FAST is a much more scalable search engine that the SharePoint native search engine, but the trade-of is an increased in complexity. FAST is quite resource hungry. However, the trade-off in search results accuracy is substantial, especially for our taxonomy navigation facets. Another feature is the more extensible pipeline. SharePoint search crawlers are not extensible like FAST crawlers are. So, there is no interface for us to link into to use BCD. We support SharePoint search only for web sites, file systems, and native SharePoint data sources.

Yes. DataFacet is a SharePoint Application Service. It requires farm administrator rights to install and run.

No. DataFacet only requires access to local resources and resources that will be indexed through the FAST or SharePoint crawlers and connectors.

It depends. We have the ability to connect to different types of search and content management systems but we need to understand how the medical library is stored/retrieved. For the most part, if a system can read a document and exposes that text via a pipeline, (SharePoint/Oracle/etc.), then the process is “I am checking in this document, and if you want to read/annotate, use this API”. We can evaluate that API against the market to see if we want to spend time and resources building an adapter for that pipeline. But, If they are just documents sitting on a file-share or web-server without some sort of centralized index them then we currently do not tag them because the tags would have to be inserted into the document to be stored somewhere and that is something that we do not want to do.

We have power shell scripts to auto re-index documents if major changes have been made.

DataFacet will overwrite all taxonomy metadata every time a document triggers an “update” event. You can manually modify the terms, but any changes to the document or metadata will only persist until the next time the document is updated.

We do In the current behavior, there is no way to permanently delete a taxonomy tag from a document. You can temporarily remove a tag at check-in time, but any subsequent update of the document will re-tag the document from the taxonomy.
A way to permanently exclude a tag from a document is to add an exclusion rule for every document that should not have the tag to the term rule in the taxonomy itself. You will need to know a unique identifier for the document (e.g, URL or Path) and add an exclusion rule. For example, if the preferred term is “Gemstone” and the rule is (gemstone OR gem OR “precious stone” OR “semiprecious stone”) and you want to exclude the document with a path \rockstore\minerals\organic\coal\anthracite.doc, you can modify the query to look like this:

(gemstone OR gem OR “precious stone” OR “semiprecious stone”) -PATH:”\rockstore\minerals\organic\coal\anthracite.doc”

Not yet. It is an active field of research. We do, however, support SharePoint Content Organizer and multiple content types. With Content Organizer, you can set up routing rules based on tags added by the DataFacet annotator. You can also assign specific taxonomy columns to specific content types to limit which content types are annotated with specific taxonomies.

I don’t believe we have tested this precise case. We do honor ACL permissions on the term store – but I think this is a more granular level.

No. All taxonomy information is added to managed metadata site columns in SharePoint. Taxonomy annotations are not propagated back to the source document properties.

1. Copy the datafacet installer exe file to each WFE and the main Application Server on the farm.
2. RDP to the server where you want the DataFacet service application running
3. double-click .exe
4. Select the Farm installation parameters
            a. Checked "Provision service application"
            b. Checked "Install service application"
            c. Select the Application Server that will host DataFacet from the Target Server      dropdown
5. For each additional server in the farm
    a. RDP to the server
    b. Double-click on the .exe
    c. Uncheck the "Provision service application" and "Install service instance" checkboxes

This simply copies code the is  required by the feature, but doesn't deploy anything to the WFE.

How do you turn tagging on for all document libraries in a site? a) There is no way to turn it on on a specific site only. You could turn it on for a specific document library however.

How do you turn it on for all document libraries in a site collection? Activate or Deactivate the Site Collection Feature: “DataFacet Annotator toggle for all Document Libraries in this Site Collection”

When you activate this feature, DataFacet will enable the Annotator for all document libraries in this site collection. You can then manually disable the Annotator on individual Document Libraries through Library Settings. When you deactivate this feature, DataFacet will disable the Annotator for all document libraries in this Site Collection. Requires the feature “DataFacet Annotator Enabled on this Site Collection to be activated.

Yes, it can. Same option is available on the List Setting page. However, one would need to have field rules (since tasks have no stream content) like “title:blah” to get it working for non-documents. (If it is not working right now – it is a bug).

The name of a file is metadata by definition, so by default all documents will have that metadata stored in the path field when it is imported. The next logical step is to classify a document according to a file name.In fact, SharePoint has that feature built-in.

http://blogs.technet.com/b/speschka/archive/2009/10/30/sharepoint-2010-content-organizer-part-1-a-cool-new-feature-for-managing-your-content.aspx http://msdn.microsoft.com/en-us/library/ee558288.aspx

DataFacet can work with Content Organizer to augment the classification with auto-tagging based on document content. Both features work together to provide the ultimate classification and tagging combo.

This account must have explicit read and write permissions to Data Facet termstore group. And that is it. Although, we are going to change this behavior in future.

Users are able to make changes to the tags that are applied by the automatic tagging engine. A user is able to delete tags or to add additional tags of their own. However, if you trigger a document set to be retagged based upon new rules or new taxonomies, the document will be re-tagged based upon the tagging rules and any manual changes made by the user will be over-ridden In future versions of DataFacet, there will be options for retaining changes made by end –users even in cases of re-tagging.

Very little impact. Our annotator is very fast, with the ability to process 200,000 queries per second on commodity hardware. In a general sense, the annotator does not contribute any perceptible overhead to the ingestion process, except possibly for the first document which populates the caches.

That is somewhat configurable. By default, we create a fairly large index by storing all document text in the Lucene index file. The reason we do this is to allow preview of results documents directly from the stored text in the index. This is configurable, however. So, if you find your Lucene index is getting too big, we can configure it to store less information and show truncated document previews in teh taxonomy manager.

In general terms, DataFacet is much lighter-weight than SharePoint. Chances are if you meet the minimum requirements to run SharePoint, you automatically have the minimum requirements to run DataFacet. The biggest variable is disk space for the intermediate Lucene index. If you have a lot of documents, you will need a fairly large disk drive to handle the full text index. Remember, however, that document text is usually much smaller than it’s container. The actual text in a 1MB PDF file is often just a few KB. So if your documents contain a lot of formatting overhead (PDF, Graphics, Multimedia), then the ratio of stored text to source document size can be quite small.

DataFacet does not use a SQL database, so there is no reason to allocate storage on a database server. We do maintain a local full-text index that is used for testing. RebelSearch uses a similar index as well, but both are self-contained data stores.

As a rule of thumb, the internal data storage will depend heavily on the character of the incoming documents. It will be some percentage of the documents in the repository depending on the ratio of non-text formatting to actual text. We only store the text part of a document, so markup and images are completely discarded. There is no way to pre-calculate the percentage – but an estimate can be made based on the length and type of documents.

PDF Images from scanned sources will have a very low text/format ratio. A 100MB file could easily have only 1k of text. Word Documents are often mostly text – so they would have a fairly high text/format ratio. A 100KB Word file might have 80KB text, if there are no images in the document. You can extrapolate to other document formats.

A safe rule-of-thumb would be to have 100% of the document source size available on indexes. So, if you have 100GB of data, you should have at least 100GB of local storage for the index.

RebelSearch crawling of about 1500 documents on commodity hardware running under a virtual machine. The mix of documents was from an internal Renaissance IT data set in English and Russian, with a fairly good mix of content types. This is a total throughput number, meaning that it includes crawling, filtering , tagging and indexing into Lucene. Alexey estimates that AR Classifier contributes about 50% of the latency in the process, meaning that without the classifier stage, the indexer is about twice as fast.

These are legitimate numbers, but they are not rigorous numbers. Also, they do not apply to SharePoint Search or FAST search, which will have completely different metrics that we haven’t tested yet. We can probably get real world numbers from IMF for indexing about 30,000 mostly PDF and Office documents on FAST, but again the numbers will include several other items including SeeUnity connectors, filtering and indexing in addition to tagging.

Theoretically, we are scalable to much better numbers simply by throwing hardware or cloud infrastructure at the problem. We also have an Azure cloud implementation fo the classifier that could be re-animated in a week or so if we got a really big sale, or needed to meet some kind of performance criteria.

In my experience with Verity and Autonomy, it is a bad idea to quote any kind of performance numbers that have not been empirically tested with real customer date on customer equipment. There is such a wide variety of document types, taxonomy sizes and hardware configurations, that it is impossible to state an actual number with any certainty, since we have not done rigorous performance testing in the SharePoint environment. DataFacet for SharePoint is slower than RebelSearch, but by how much depends on many factors.

If we want to get real numbers, it must be a project with resources allocated and a compelling business reason to allocate those resources. Performance metrics can be very difficult to model outside of a real-world implementation.

In my opinion we should be very circumspect about throwing numbers like “40 documents per second” or even our 200,000 rules per second metric. We have proven both in certain contexts – but things could go very wrong if we start committing to any particular number. A better approach is to ask the customer what their performance requirements are and ensure that we side the implementation accordingly, building in time for benchmarking if necessary.

What we can say is that we are damn fast, and that we are damn scalable, and we’re up to meeting any performance metric challenge that we get.

DataFacet is designed to run on existing SharePoint hardware and has a minimal impact on SharePoint performance. Ongoing day-to-day document check-in should not cause any noticeable change to SharePoint performance. If you plan to tag a large number of documents at a time, it is advisable to do this over a weekend or during other off-peak hours.

EXAMPLE: What if one of their medical clients want to tag head and neck neoplasm’s. They want head and neck to be in the same sentence and neoplasm to be in the title.

At the very least, can the tag be designated as title specific? Proximity searches are available to some extent (not as much ‘same sentence’ as within x characters). Both proximity (“head neoplasm”~10 OR “neck neoplasm”~10) and Field (title: “head neoplasm”~10) are supported.

Currently changes to one translation of a taxonomy will not change the translation of the term in another taxonomy as they are separate objects.

Currently no,

We do not model the “Reuse” option, because that is essentially a “reference” term that we do not currently support in our UI. Each term is unique, even if it shares a name with another term. For examples; /Animals/Bears is completely different from /Football Teams/Bears. Even though the name “Bears” is the same, they are completely different objects.

A user may wish to use the proximity operators. Is there something that is changed about the query language that hides the proximity ~n operator?

Example I search for “Motion during the 2011 year” – 1 result

but…

“Motion 2011 year”~8 – 0 results.

Is the proximity operator working as intended or is something re-writing it?

DataFacet supports finding words are a within a specific distance away. To do a proximity search use the tilde, “~”, symbol at the end of a Phrase. For example to search for a “bean” and “coffee” within 10 words of each other in a document use the search:

 “coffee bean”~10

Using DataFacet, can files that are named based on the type of content in each can these documents be imported to SharePoint so that they are applied as metadata to each document, knowing that there is no taxonomy beyond the file name?

The name of a file is metadata by definition, so by default all documents will have that metadata stored in the path field when it is imported.The next logical step is to classify a document according to a file name. In fact, SharePoint has that feature built-in.

http://blogs.technet.com/b/speschka/archive/2009/10/30/sharepoint-2010-content-organizer-part-1-a-cool-new-feature-for-managing-your-content.aspx http://msdn.microsoft.com/en-us/library/ee558288.aspx

DataFacet can work with Content Organizer to augment the classification with auto-tagging based on document content. Both features work together to provide the ultimate classification and tagging combo.

Yes, taxonomies are available in multiple languages, including German, Spanish, French, Italian, Portuguese, Traditional Chinese, Simplified Chinese, Japanese, Korean, and Vietnamese. Limited parts of the taxonomies are translated into Swedish and Finnish.  Taxonomies can be translated into any languages quickly (using human translators).

Yes, Re-used terms are updated across term groups. However, it should be noted that adding a narrower term to one does not automatically add the same narrower term to another. Only the term itself is synchronized. Also, be careful not to confuse copying a term with reusing a term.

“To restore the DataFacet Term group:” 1. Open your term store management page. (Central Administration -> Manage Service Applications -> Managed Metadata Service)

1. Expand Managed Metadata Service context menu, click New Group
2. Enter “”"”Data Facet”"”" without double quotes & hit Enter
3. Ensure that Group Managers has at least this accounts: account used to run Central Admin app pool, account used to run DataFacet taxonomy service app pool, your account.
4. run iisreset
5. Check the “”"”manage taxonomies”"”" page to ensure that you could see an empty list of taxonomies now without any errors “”"

Check to make sure your operator is all uppercase. Otherwise it will be treated as a stop word.

The Adobe web site seems to suggest that the stand-alone iFilter is not required if you install the latest Acrobat Reader. This is true for the desktop search, but for server products like SharePoint, you still must install the Adobe iFilter for 64bit platforms.

http://www.adobe.com/support/downloads/thankyou.jsp?ftpID=4025&fileID=3941

If not, application pool will fail to start and there will be error log entry in Event Log. Something like: …this pool failed to start because login credentials are invalid…

It depends. First if DataFacet uses a different account to run service application – no effect. Second, if password was updated through SharePoint interface – no issues too. The only problem could be if password changed outside of SharePoint (and IIS), then IIS’s Application Pool will fail start.

This issue sometimes appear after product upgrade. Basically, .webpart files stored in WebParts Gallery get outdated. In order to upgrade them you will need to:

1. Site Actions -> Site Settings
2. Click on “Web parts” under “Galleries”
3. Find & delete following files CurrentRefinementConnectionProvider.webpart CurrentRefinementsWebPart.webpart TaxonomyKeywordListRefinementWebPart.webpart TaxonomyTreeRefinementWebPart.webpart
4. Return back to Site Collection Settings page
5. Click on “Site collection features” under “Site Collection Administration”
6. Reactivate (deactivate & active) the feature called “DataFacet Annotator enabled on this Site Collection”

That should force new webpart files to be uploaded into the WebPart gallery & resolve the deserialization issue.

The error you saw is not entirely uncommon. We’re working on a troubleshooting guide to address it, but in the mean time – here are some steps you can take on your own to troubleshoot.

The “Error 503″ refers to the fact that the DataFacet web service application is not available for some reason. The most common reason is that the application pool associated with the web service application cannot be started because of invalid credentials. If you installed DataFacet under a specific account and then later changed the password or privileges of that account, then the DataFacet application service will not start because it’s application pool will not start.

1) Make note of the time. If you can afford the luxury, you may want to clear out the system and application event logs to make it easier to spot error messages – but it is not necessary at all.
2) Run IISRESET to try to restart all of the SharePoint application services.
3) After IISRESET has run, make sure that SharePoint Central Administration and the Managed Metadata service are both up and running by visiting them in the browser.
4) Click on the DataFacet service application link. If the page shows, you probably just needed a remedial IISRESET and you are good to go.
5) If you get the 503 error again, go to the event log and look for a succession of two warnings and an error from the WAS Source:

The error will be something like “Application pool xxxxxxxxxxxxxxxxxxxxxxxx has been disabled. Windows Process Activation Service (WAS) encountered a failure when it started a worker process to serve the application pool”
The two warnings will hint at why the application pool failed to start – probably having to do with authentication.

6) Now that you know the identity of the application pool (memorize the first few characters so it is easy to find), we can visit IIS Administration and verify that the Application Pool is indeed stopped.

7) To change the application pools credentials, select the application pool and click on “Advanced Settings”. This will bring up a dialog box with application pool properties.

Under the Process Model section, click on the [...] icon to change the Identity value. Enter valid credentials for the application pool user. The application pool user must have read-write access to the SharePoint term store, specifically the Data Facet term store group.

After that, DataFacet should work again.

Yes. We have a Stand-Alone HTML5 application that can be hosted in Adobe Air or on any modernbrowser that supports HTML 5. It has the same features as the Central Admin Version, and is basedon the same JavaScript code base.